[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:20 dist-test-slave-3rxd krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:20 dist-test-slave-3rxd krb5kdc[395](info): set up 2 sockets
Jun 19 00:59:20 dist-test-slave-3rxd krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 00:59:21.323843 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.010s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:21 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:21.494988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35559
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:21.827863 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:21.828148 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:21.828313 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:21.835530 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:21.835629 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:21.835757 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:21.835841 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:21.835923 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:21.846342 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:21.847774 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:21.849114 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:21.861253 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:21.861322 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:21.861285 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:21.862080 408 server_base.cc:1034] running on GCE node
I20260619 00:59:21.862886 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:21.864185 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:21.865423 408 hybrid_clock.cc:648] HybridClock initialized: now 1781830761865365 us; error 71 us; skew 500 ppm
I20260619 00:59:21.865767 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:21 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830761, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:21.868781 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:21.870172 408 webserver.cc:460] Webserver started at http://127.0.97.62:40045/ using document root <none> and password file <none>
I20260619 00:59:21.870935 408 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:21.871071 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:21.871362 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:21.874408 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "eff8986a24a2476b844f8677e1351dff"
format_stamp: "Formatted at 2026-06-19 00:59:21 on dist-test-slave-3rxd"
server_key: "6085c7e519672b1ad065c47647b4c641"
server_key_iv: "7f102fd7a6556780049f19259b052948"
server_key_version: "encryptionkey@0"
I20260619 00:59:21.875151 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "eff8986a24a2476b844f8677e1351dff"
format_stamp: "Formatted at 2026-06-19 00:59:21 on dist-test-slave-3rxd"
server_key: "6085c7e519672b1ad065c47647b4c641"
server_key_iv: "7f102fd7a6556780049f19259b052948"
server_key_version: "encryptionkey@0"
I20260619 00:59:21.880095 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 00:59:21.883419 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:21.884629 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:59:21.884809 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "eff8986a24a2476b844f8677e1351dff"
format_stamp: "Formatted at 2026-06-19 00:59:21 on dist-test-slave-3rxd"
server_key: "6085c7e519672b1ad065c47647b4c641"
server_key_iv: "7f102fd7a6556780049f19259b052948"
server_key_version: "encryptionkey@0"
I20260619 00:59:21.885023 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:21.900202 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:21.904219 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:21.904455 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:21.923673 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35559
I20260619 00:59:21.924119 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35559 every 8 connection(s)
I20260619 00:59:21.925226 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:21.928704 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:21.932183 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 408
I20260619 00:59:21.932497 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:21.934909 388 external_mini_cluster.cc:1428] Setting key 4aafedcf334d0130fa4fee5c6d9eec6b
I20260619 00:59:21.940932 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff: Bootstrap starting.
I20260619 00:59:21.943775 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:21.944725 475 log.cc:824] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:21.947310 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff: No bootstrap required, opened a new log
I20260619 00:59:21.956735 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } }
I20260619 00:59:21.957154 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:21.957262 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8986a24a2476b844f8677e1351dff, State: Initialized, Role: FOLLOWER
I20260619 00:59:21.957861 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } }
I20260619 00:59:21.958092 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:21.958240 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:21.958410 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:21.961455 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } }
I20260619 00:59:21.961974 475 leader_election.cc:302] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eff8986a24a2476b844f8677e1351dff; no voters:
I20260619 00:59:21.962634 475 leader_election.cc:288] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:21.962770 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:21.964591 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [term 1 LEADER]: Becoming Leader. State: Replica: eff8986a24a2476b844f8677e1351dff, State: Running, Role: LEADER
I20260619 00:59:21.965106 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } }
I20260619 00:59:21.965600 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 00:59:21 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:21.968014 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eff8986a24a2476b844f8677e1351dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } } }
I20260619 00:59:21.968464 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:21.972479 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader eff8986a24a2476b844f8677e1351dff. Latest consensus state: current_term: 1 leader_uuid: "eff8986a24a2476b844f8677e1351dff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eff8986a24a2476b844f8677e1351dff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35559 } } }
I20260619 00:59:21.972754 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:21.975670 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:21.980019 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:21.939261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.0.97.62:35559)
0619 00:59:21.940098 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:21.940112 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:21.941803 (+ 1691us) server_negotiation.cc:407] Connection header received
0619 00:59:21.952347 (+ 10544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:21.952379 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:21.952858 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:21.953275 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:21.955903 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:21.957769 (+ 1866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:21.959743 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:21.961184 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:21.961376 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:21.968077 (+ 6701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:21.968091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:21.968108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:21.968158 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:21.973595 (+ 5437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:21.974810 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:21.974818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:21.974825 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:21.975032 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:21.975661 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:21.975668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:21.975671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:21.976364 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:21.976655 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:21.977865 (+ 1210us) server_negotiation.cc:299] Negotiation successful
0619 00:59:21.978386 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":370,"threads_started":1}
I20260619 00:59:21.984094 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:21.992594 485 catalog_manager.cc:1347] Generated new cluster ID: f221eeacec754a9da1ee2c80b47b7b63
I20260619 00:59:21.992703 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:22.005069 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:22.006822 485 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:22.015475 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eff8986a24a2476b844f8677e1351dff: Generated new TSK 0
I20260619 00:59:22.016291 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:22.064390 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:22.395452 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:22.395753 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:22.395926 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:22.403038 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:22.403182 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:22.403357 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:22.413538 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:22.414937 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:22.416244 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:22.429277 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:22.429251 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:22.429224 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:22.430037 501 server_base.cc:1034] running on GCE node
I20260619 00:59:22.430657 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:22.431537 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:22.432758 501 hybrid_clock.cc:648] HybridClock initialized: now 1781830762432720 us; error 52 us; skew 500 ppm
I20260619 00:59:22.433060 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:22 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:22.436156 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:22.437497 501 webserver.cc:460] Webserver started at http://127.0.97.1:39153/ using document root <none> and password file <none>
I20260619 00:59:22.438148 501 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:22.438270 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:22.438570 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:22.441416 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "2b0bedcb14a1401a84bccbfacce8ca74"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "d6ca675d10f1ba3d0c606b29f7850229"
server_key_iv: "ff095560fcabce6332af218eb75fa0c2"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.442169 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2b0bedcb14a1401a84bccbfacce8ca74"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "d6ca675d10f1ba3d0c606b29f7850229"
server_key_iv: "ff095560fcabce6332af218eb75fa0c2"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.447432 501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 00:59:22.450582 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:22.451874 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:59:22.452032 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "2b0bedcb14a1401a84bccbfacce8ca74"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "d6ca675d10f1ba3d0c606b29f7850229"
server_key_iv: "ff095560fcabce6332af218eb75fa0c2"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.452229 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:22.476044 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:22.480187 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:22.480489 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:22.481505 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:22.483203 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:22.483302 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:22.483446 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:22.483515 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:22.518216 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41107
I20260619 00:59:22.518246 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41107 every 8 connection(s)
I20260619 00:59:22.519605 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:22.521836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 501
I20260619 00:59:22.522063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:22.522656 388 external_mini_cluster.cc:1428] Setting key fce04d773adb9017264a4103ddaf2803
Jun 19 00:59:22 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:59:22.540557 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:22.523162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54333 (local address 127.0.97.62:35559)
0619 00:59:22.524352 (+ 1190us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:22.524358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:22.524396 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:59:22.525863 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:22.525869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:22.525959 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:22.526087 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:22.527698 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:22.528433 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:22.530001 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:22.530409 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:22.530549 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:22.533894 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:22.533903 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:22.533912 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:22.533941 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:22.537087 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:22.537904 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:22.537910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:22.537914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:22.538039 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:22.538565 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:22.538570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:22.538574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:22.538945 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:22.539105 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:22.539901 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 00:59:22.540134 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":977,"threads_started":1}
I20260619 00:59:22.541587 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:22.522665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35559 (local address 127.0.97.1:54333)
0619 00:59:22.523560 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:22.523638 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:22.525573 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:22.526342 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:22.526351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:22.526942 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:22.527524 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:22.527546 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:22.528587 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:22.528597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:22.529857 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:22.529868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:22.530637 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:22.530645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:22.530919 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:22.531610 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:22.531633 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:22.533676 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:22.537253 (+ 3577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:22.537263 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:22.537280 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:22.537769 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:22.538154 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:22.538160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:22.538164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:22.538431 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:22.539095 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:22.539103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:22.539464 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 00:59:22.539756 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 00:59:22.540352 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":409,"threads_started":1}
I20260619 00:59:22.543323 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35559
I20260619 00:59:22.543666 629 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:22.544361 629 heartbeater.cc:511] Master 127.0.97.62:35559 requested a full tablet report, sending...
I20260619 00:59:22.546694 440 ts_manager.cc:194] Registered new tserver with Master: 2b0bedcb14a1401a84bccbfacce8ca74 (127.0.97.1:41107)
I20260619 00:59:22.548882 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:22.561900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:22.885102 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:22.885372 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:22.885466 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:22.892766 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:22.892872 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:22.892985 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:22.902864 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:22.904071 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:22.905309 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:22.919903 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:22.919948 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:22.920145 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:22.920588 635 server_base.cc:1034] running on GCE node
I20260619 00:59:22.921175 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:22.922147 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:22.923359 635 hybrid_clock.cc:648] HybridClock initialized: now 1781830762923320 us; error 62 us; skew 500 ppm
I20260619 00:59:22.923725 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:22 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:22.930691 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:22.932236 635 webserver.cc:460] Webserver started at http://127.0.97.2:42651/ using document root <none> and password file <none>
I20260619 00:59:22.932898 635 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:22.933014 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:22.933316 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:22.936177 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "a043158bcd2743b396facfe05680a21f"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "2928d206a13520cd628eb73ca7aa234c"
server_key_iv: "83caebd8cd67ffdc0c569cc5d1daab1b"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.936945 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a043158bcd2743b396facfe05680a21f"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "2928d206a13520cd628eb73ca7aa234c"
server_key_iv: "83caebd8cd67ffdc0c569cc5d1daab1b"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.943218 635 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260619 00:59:22.948200 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:22.949671 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 00:59:22.949900 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "a043158bcd2743b396facfe05680a21f"
format_stamp: "Formatted at 2026-06-19 00:59:22 on dist-test-slave-3rxd"
server_key: "2928d206a13520cd628eb73ca7aa234c"
server_key_iv: "83caebd8cd67ffdc0c569cc5d1daab1b"
server_key_version: "encryptionkey@0"
I20260619 00:59:22.950157 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:22.997486 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:23.001425 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:23.001708 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:23.002653 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:23.003937 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:23.004035 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:23.004177 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:23.004249 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:23.039191 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41113
I20260619 00:59:23.039273 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41113 every 8 connection(s)
I20260619 00:59:23.040494 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:23.048923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 635
I20260619 00:59:23.049106 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:23.049551 388 external_mini_cluster.cc:1428] Setting key 0302f82c8b1f0ae748a49d168d800966
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:23.062464 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.043079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59335 (local address 127.0.97.62:35559)
0619 00:59:23.043490 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.043496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.044889 (+ 1393us) server_negotiation.cc:407] Connection header received
0619 00:59:23.046615 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.046619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.046691 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.046844 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.048214 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.049117 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.050697 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.051180 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.051316 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.055499 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.055504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.055507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.055538 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.059367 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.060059 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.060068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.060074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.060220 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.060598 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.060605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.060611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.060978 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.061129 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.061746 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.061990 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":189,"threads_started":1}
I20260619 00:59:23.063189 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.043655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35559 (local address 127.0.97.2:59335)
0619 00:59:23.044718 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:23.044755 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:23.046370 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:23.047048 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:23.047056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:23.047551 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:23.048073 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:23.048089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.049278 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.049281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:23.050545 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:23.050556 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.051406 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.051413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:23.051624 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.052293 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:23.052314 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:23.055264 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:23.059488 (+ 4224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:23.059494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:23.059507 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:23.059943 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:23.060324 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:23.060327 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:23.060329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:23.060493 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:23.061107 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:23.061112 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:23.061391 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 00:59:23.061603 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 00:59:23.061979 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":667,"threads_started":1}
I20260619 00:59:23.064582 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35559
I20260619 00:59:23.064913 763 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:23.065807 763 heartbeater.cc:511] Master 127.0.97.62:35559 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:59:23.067819 440 ts_manager.cc:194] Registered new tserver with Master: a043158bcd2743b396facfe05680a21f (127.0.97.2:41113)
I20260619 00:59:23.068684 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:23.090189 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:23.425374 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:23.425658 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:23.425835 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:23.433118 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:23.433274 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:23.433473 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:23.443181 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35559
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:23.444468 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:23.445827 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:23.459787 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:23.459887 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:23.459815 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:23.460680 769 server_base.cc:1034] running on GCE node
I20260619 00:59:23.461264 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:23.462138 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:23.463368 769 hybrid_clock.cc:648] HybridClock initialized: now 1781830763463312 us; error 45 us; skew 500 ppm
I20260619 00:59:23.463673 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:23.467242 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:23.468645 769 webserver.cc:460] Webserver started at http://127.0.97.3:46743/ using document root <none> and password file <none>
I20260619 00:59:23.469318 769 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:23.469468 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:23.469779 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:23.472671 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "5fdd2fdbec67417aa5feb79058b708c0"
format_stamp: "Formatted at 2026-06-19 00:59:23 on dist-test-slave-3rxd"
server_key: "810a29ee0f39e67e32abadda30dcbeea"
server_key_iv: "e24fee9658baf43f4855a33db1150fc4"
server_key_version: "encryptionkey@0"
I20260619 00:59:23.473465 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "5fdd2fdbec67417aa5feb79058b708c0"
format_stamp: "Formatted at 2026-06-19 00:59:23 on dist-test-slave-3rxd"
server_key: "810a29ee0f39e67e32abadda30dcbeea"
server_key_iv: "e24fee9658baf43f4855a33db1150fc4"
server_key_version: "encryptionkey@0"
I20260619 00:59:23.478415 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:59:23.481696 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:23.483027 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:23.483191 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "5fdd2fdbec67417aa5feb79058b708c0"
format_stamp: "Formatted at 2026-06-19 00:59:23 on dist-test-slave-3rxd"
server_key: "810a29ee0f39e67e32abadda30dcbeea"
server_key_iv: "e24fee9658baf43f4855a33db1150fc4"
server_key_version: "encryptionkey@0"
I20260619 00:59:23.483393 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:23.496474 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:23.500178 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:23.500464 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:23.501344 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:23.502586 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:23.502687 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:23.502866 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:23.502940 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:23.543118 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38619
I20260619 00:59:23.543148 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38619 every 8 connection(s)
I20260619 00:59:23.544378 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:23.546096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 769
I20260619 00:59:23.546278 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:23.546693 388 external_mini_cluster.cc:1428] Setting key ab2003c42513cc54188187f01af694c0
I20260619 00:59:23.552708 629 heartbeater.cc:503] Master 127.0.97.62:35559 was elected leader, sending a full tablet report...
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:23.564930 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.547092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55207 (local address 127.0.97.62:35559)
0619 00:59:23.547315 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.547323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.549107 (+ 1784us) server_negotiation.cc:407] Connection header received
0619 00:59:23.551208 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.551212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.551273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.551353 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.552851 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.553973 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.555782 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.556157 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.556245 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.559364 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.559369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.559372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.559400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.561938 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.562586 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.562591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.562593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.562699 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.563113 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.563116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.563118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.563366 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.563466 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.564153 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.564302 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:59:23.565490 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.547778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35559 (local address 127.0.97.3:55207)
0619 00:59:23.548553 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:23.548605 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:23.550970 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:23.551559 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:23.551566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:23.552106 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:23.552665 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:23.552682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.554134 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.554143 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:23.555629 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:23.555641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.556337 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.556341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:23.556524 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.557106 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:23.557126 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:23.559174 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:23.562053 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:23.562061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:23.562077 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:23.562477 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:23.562837 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:23.562840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:23.562842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:23.563015 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:23.563460 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:23.563465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:23.563813 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 00:59:23.564064 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 00:59:23.564410 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":327,"threads_started":1}
I20260619 00:59:23.566967 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35559
I20260619 00:59:23.567266 897 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:23.567916 897 heartbeater.cc:511] Master 127.0.97.62:35559 requested a full tablet report, sending...
I20260619 00:59:23.569200 440 ts_manager.cc:194] Registered new tserver with Master: 5fdd2fdbec67417aa5feb79058b708c0 (127.0.97.3:38619)
I20260619 00:59:23.569968 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55207
I20260619 00:59:23.574164 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:23.647743 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.631636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.0.97.1:41107)
0619 00:59:23.632444 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.632451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.632479 (+ 28us) server_negotiation.cc:407] Connection header received
0619 00:59:23.632542 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.632549 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.632756 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.632912 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.634353 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.635433 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.636649 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.637226 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.637322 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.641474 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.641485 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.641501 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.641539 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.644292 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.644946 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.644953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.644955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.645082 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.646188 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.646192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.646194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.646536 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.646844 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.647121 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.647331 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":502,"threads_started":1}
W20260619 00:59:23.649024 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50424
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:23.724344 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.709733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50440 (local address 127.0.97.1:41107)
0619 00:59:23.709945 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.709952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.710233 (+ 281us) server_negotiation.cc:407] Connection header received
0619 00:59:23.710413 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.710417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.710506 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.710594 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.712512 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.713185 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.714481 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.715230 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.715362 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.719546 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.719552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.719562 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.719594 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.721853 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.722459 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.722466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.722470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.722592 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.722991 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.722998 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.723002 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.723252 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.723507 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.723797 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.724012 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 00:59:23.725852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 501
I20260619 00:59:23.738039 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.726813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.0.97.1:41107)
0619 00:59:23.727020 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.727028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.727315 (+ 287us) server_negotiation.cc:407] Connection header received
0619 00:59:23.727422 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.727425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.727476 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.727551 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.728600 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.729256 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.730544 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.731012 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.731103 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.733339 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.733343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.733347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.733380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.735867 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.736447 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.736450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.736452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.736581 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.736921 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.736924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.736926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.737160 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.737326 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.737564 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.737707 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 00:59:23.738768 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:23.822921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 501
I20260619 00:59:23.838878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 635
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:23.855504 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.839758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.0.97.2:41113)
0619 00:59:23.840355 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.840362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.840388 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:59:23.840444 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.840452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.840660 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.840793 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.841837 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.842969 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.844106 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.844698 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.844839 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.849264 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.849277 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.849293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.849329 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.852340 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.853051 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.853058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.853061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.853176 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.853677 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.853680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.853682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.854023 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.854237 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.854666 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.854994 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":328,"threads_started":1}
I20260619 00:59:23.856737 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:23.941728 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 635
I20260619 00:59:23.965014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 769
Jun 19 00:59:23 dist-test-slave-3rxd krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:23.981777 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:23.966084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.97.3:38619)
0619 00:59:23.966662 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:23.966668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:23.966698 (+ 30us) server_negotiation.cc:407] Connection header received
0619 00:59:23.966814 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:23.966820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:23.967121 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:23.967270 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:23.968523 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.969646 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.971053 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:23.971585 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:23.971694 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:23.975967 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:23.975978 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:23.975994 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:23.976038 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:23.978681 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.979615 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.979631 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.979634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.979754 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:23.980189 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:23.980196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:23.980200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:23.980579 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:23.980932 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:23.981170 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 00:59:23.981443 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":290,"threads_started":1}
I20260619 00:59:23.983121 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:24.074563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 769
I20260619 00:59:24.090888 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 408
I20260619 00:59:24.092072 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:24.193343 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 408
2026-06-19T00:59:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3970 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:26 dist-test-slave-3rxd krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:26 dist-test-slave-3rxd krb5kdc[918](info): set up 2 sockets
Jun 19 00:59:26 dist-test-slave-3rxd krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260619 00:59:27.528791 388 mini_kdc.cc:121] Time spent starting KDC: real 3.293s user 0.008s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 00:59:27.694977 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.166s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 00:59:27.798998 388 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.104s user 0.002s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/test-user.keytab.
W20260619 00:59:28.753062 388 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.868s user 0.001s sys 0.004s
Jun 19 00:59:28 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
W20260619 00:59:28.981058 388 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.62: real 0.184s user 0.000s sys 0.007s
I20260619 00:59:28.982098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39819
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:29.300345 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:29.300608 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:29.300694 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:29.307732 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:29.307861 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:29.307911 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:29.307968 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:29.308007 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:29.317857 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:29.319113 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:29.320339 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:29.331848 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:29.331766 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:29.331809 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:29.332713 931 server_base.cc:1034] running on GCE node
I20260619 00:59:29.333454 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:29.334904 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:29.336119 931 hybrid_clock.cc:648] HybridClock initialized: now 1781830769336076 us; error 52 us; skew 500 ppm
I20260619 00:59:29.336443 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:29 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:29.339515 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:29.341010 931 webserver.cc:460] Webserver started at http://127.0.97.62:41493/ using document root <none> and password file <none>
I20260619 00:59:29.341691 931 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:29.341823 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:29.342133 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:29.345090 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "67695e78f6504ccb805443dab085b16a"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "ac92efa6350a4670988b6f54f34d6c81"
server_key_iv: "c5d19b7ff51138cd2eab280515589c8f"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.345849 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "67695e78f6504ccb805443dab085b16a"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "ac92efa6350a4670988b6f54f34d6c81"
server_key_iv: "c5d19b7ff51138cd2eab280515589c8f"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.351128 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 00:59:29.354441 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:29.355731 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:29.355916 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "67695e78f6504ccb805443dab085b16a"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "ac92efa6350a4670988b6f54f34d6c81"
server_key_iv: "c5d19b7ff51138cd2eab280515589c8f"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.356102 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:29.380101 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:29.384101 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:29.384372 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:29.401841 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39819
I20260619 00:59:29.401925 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39819 every 8 connection(s)
I20260619 00:59:29.403208 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:29.406883 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:29.410943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 931
I20260619 00:59:29.411167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:29.411993 388 external_mini_cluster.cc:1428] Setting key 86b8c58c1f206c5ab2a1457ed96746ab
I20260619 00:59:29.418186 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a: Bootstrap starting.
I20260619 00:59:29.421124 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:29.422101 998 log.cc:824] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:29.425398 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a: No bootstrap required, opened a new log
Jun 19 00:59:29 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:29.433171 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } }
I20260619 00:59:29.433557 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:29.433689 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67695e78f6504ccb805443dab085b16a, State: Initialized, Role: FOLLOWER
I20260619 00:59:29.434338 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } }
I20260619 00:59:29.434587 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:29.434767 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:29.434834 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:29.414011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.97.62:39819)
0619 00:59:29.414694 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:29.414762 (+ 68us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:29.414809 (+ 47us) server_negotiation.cc:407] Connection header received
0619 00:59:29.416918 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:29.416940 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:29.417315 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:29.417703 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:29.419210 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.420498 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.421753 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.422461 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.422672 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:29.427122 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:29.427129 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:29.427141 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:29.427187 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:29.430205 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:29.430946 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:29.430960 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:29.430965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:29.431129 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:29.431506 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:29.431513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:29.431515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:29.432007 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:29.432368 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:29.432830 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 00:59:29.433258 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":310,"threads_started":1}
I20260619 00:59:29.435009 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:29.437570 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } }
I20260619 00:59:29.438148 998 leader_election.cc:302] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67695e78f6504ccb805443dab085b16a; no voters:
I20260619 00:59:29.438683 998 leader_election.cc:288] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:29.438838 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:29.439558 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [term 1 LEADER]: Becoming Leader. State: Replica: 67695e78f6504ccb805443dab085b16a, State: Running, Role: LEADER
I20260619 00:59:29.440037 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } }
I20260619 00:59:29.440577 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:29.442917 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67695e78f6504ccb805443dab085b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } } }
I20260619 00:59:29.442850 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67695e78f6504ccb805443dab085b16a. Latest consensus state: current_term: 1 leader_uuid: "67695e78f6504ccb805443dab085b16a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67695e78f6504ccb805443dab085b16a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39819 } } }
I20260619 00:59:29.443234 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:29.443233 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:29.444796 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:29.449473 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:29.458941 1008 catalog_manager.cc:1347] Generated new cluster ID: 44bbbbf98aa849fcb219d9e2abbe9a7a
I20260619 00:59:29.459060 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:29.467581 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:29.468835 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:29.476392 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67695e78f6504ccb805443dab085b16a: Generated new TSK 0
I20260619 00:59:29.477145 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:29.519090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:29.859674 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:29.859943 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:29.860131 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:29.867089 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:29.867213 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:29.867408 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:29.877030 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:29.878244 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:29.879557 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:29.891947 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:29.892048 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:29.892064 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:29.892930 1024 server_base.cc:1034] running on GCE node
I20260619 00:59:29.893517 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:29.894416 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:29.895632 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781830769895596 us; error 49 us; skew 500 ppm
I20260619 00:59:29.895933 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:29 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:29.899000 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:29.900358 1024 webserver.cc:460] Webserver started at http://127.0.97.1:32837/ using document root <none> and password file <none>
I20260619 00:59:29.901046 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:29.901638 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:29.902081 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:29.904911 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "ede99edf3fbb4f448f20c3c53a2cbe15"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "a0c237de3668e27feb737412e1a61577"
server_key_iv: "cdb4ce089adfc2ecaa9e0351fbfc7b22"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.905689 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ede99edf3fbb4f448f20c3c53a2cbe15"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "a0c237de3668e27feb737412e1a61577"
server_key_iv: "cdb4ce089adfc2ecaa9e0351fbfc7b22"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.910401 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 00:59:29.913548 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:29.914826 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:29.914981 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "ede99edf3fbb4f448f20c3c53a2cbe15"
format_stamp: "Formatted at 2026-06-19 00:59:29 on dist-test-slave-3rxd"
server_key: "a0c237de3668e27feb737412e1a61577"
server_key_iv: "cdb4ce089adfc2ecaa9e0351fbfc7b22"
server_key_version: "encryptionkey@0"
I20260619 00:59:29.915163 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:29.936818 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:29.942098 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:29.942382 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:29.943399 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:29.944859 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:29.944969 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:29.945092 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:29.945173 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:29.974990 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35337
I20260619 00:59:29.975039 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35337 every 8 connection(s)
I20260619 00:59:29.976377 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:29.985246 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1024
I20260619 00:59:29.985451 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:29.986106 388 external_mini_cluster.cc:1428] Setting key 8ae81df41c42c855c1595e38cb8c3f5d
Jun 19 00:59:29 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:29.996134 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:29.978945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52401 (local address 127.0.97.62:39819)
0619 00:59:29.979330 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:29.979335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:29.980384 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 00:59:29.982078 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:29.982082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:29.982149 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:29.982244 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:29.983686 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.984381 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.985973 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.986602 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.986776 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:29.990206 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:29.990215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:29.990223 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:29.990267 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:29.993145 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:29.993838 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:29.993843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:29.993847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:29.993965 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:29.994370 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:29.994374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:29.994378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:29.994630 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:29.994816 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:29.995481 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 00:59:29.995776 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":197,"threads_started":1}
I20260619 00:59:29.997090 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:29.979565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39819 (local address 127.0.97.1:52401)
0619 00:59:29.980227 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:29.980269 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:29.981841 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:29.982443 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:29.982450 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:29.983005 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:29.983546 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:29.983563 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.984502 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.984505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:29.985835 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:29.985846 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:29.986861 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:29.986869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:29.987117 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:29.987854 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:29.987886 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:29.989999 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:29.993281 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:29.993291 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:29.993306 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:29.993715 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:29.994072 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:29.994077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:29.994081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:29.994254 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:29.994778 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:29.994786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:29.995113 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 00:59:29.995362 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 00:59:29.996038 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260619 00:59:29.998595 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39819
I20260619 00:59:29.998908 1152 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:29.999593 1152 heartbeater.cc:511] Master 127.0.97.62:39819 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:59:30.001865 963 ts_manager.cc:194] Registered new tserver with Master: ede99edf3fbb4f448f20c3c53a2cbe15 (127.0.97.1:35337)
I20260619 00:59:30.003485 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:30.025547 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:30.333362 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:30.333636 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:30.333731 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:30.341605 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:30.341755 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:30.341892 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:30.351744 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:30.353024 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:30.354293 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:30.367144 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:30.367208 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:30.367267 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:30.368225 1158 server_base.cc:1034] running on GCE node
I20260619 00:59:30.368849 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:30.369673 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:30.370862 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781830770370818 us; error 46 us; skew 500 ppm
I20260619 00:59:30.371248 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:30 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:30.374112 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:30.375438 1158 webserver.cc:460] Webserver started at http://127.0.97.2:39217/ using document root <none> and password file <none>
I20260619 00:59:30.376027 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:30.376129 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:30.376372 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:30.379113 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "74bcbd5e0a264c7094eaf28167c05e17"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "35a837785d77023148110afc8f15d3cb"
server_key_iv: "7b6df518ff570efead300283d3b2ce6b"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.379802 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "74bcbd5e0a264c7094eaf28167c05e17"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "35a837785d77023148110afc8f15d3cb"
server_key_iv: "7b6df518ff570efead300283d3b2ce6b"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.384403 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 00:59:30.387524 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.388752 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:30.388870 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "74bcbd5e0a264c7094eaf28167c05e17"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "35a837785d77023148110afc8f15d3cb"
server_key_iv: "7b6df518ff570efead300283d3b2ce6b"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.389022 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:30.402495 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:30.406483 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:30.406822 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:30.407752 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:30.409016 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:30.409085 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.409240 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:30.409296 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.441663 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46637
I20260619 00:59:30.441696 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46637 every 8 connection(s)
I20260619 00:59:30.443264 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:30.450649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1158
I20260619 00:59:30.450865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:30.451375 388 external_mini_cluster.cc:1428] Setting key 1f821d52775d281b623b20d6a53ff9e1
Jun 19 00:59:30 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:30.463752 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.445697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42069 (local address 127.0.97.62:39819)
0619 00:59:30.445899 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.445905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:30.447066 (+ 1161us) server_negotiation.cc:407] Connection header received
0619 00:59:30.448751 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:30.448755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:30.448822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:30.448909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:30.450327 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.451525 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.453425 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.454052 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.454203 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:30.458027 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:30.458032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:30.458036 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:30.458066 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:30.460820 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.461514 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.461522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.461525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.461648 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.462060 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.462067 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.462070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.462325 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:30.462494 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:30.463196 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 00:59:30.463409 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:59:30.464773 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.446181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39819 (local address 127.0.97.2:42069)
0619 00:59:30.446918 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:30.446959 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:30.448507 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:30.449115 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:30.449123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:30.449618 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:30.450166 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.450182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.451683 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.451706 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:30.453286 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.453299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.454244 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.454248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:30.454545 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:30.455279 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:30.455301 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:30.457813 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:30.460967 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:30.460976 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:30.460993 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:30.461390 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:30.461754 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:30.461760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:30.461764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:30.461945 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:30.462450 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:30.462459 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:30.462848 (+ 389us) client_negotiation.cc:769] Sending connection context
0619 00:59:30.463119 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 00:59:30.463636 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":271,"threads_started":1}
I20260619 00:59:30.466358 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39819
I20260619 00:59:30.466661 1286 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:59:30.467381 1286 heartbeater.cc:511] Master 127.0.97.62:39819 requested a full tablet report, sending...
I20260619 00:59:30.468768 963 ts_manager.cc:194] Registered new tserver with Master: 74bcbd5e0a264c7094eaf28167c05e17 (127.0.97.2:46637)
I20260619 00:59:30.469660 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:30.490121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:30.799559 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:30.799814 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:30.799993 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:30.807070 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:30.807195 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:30.807382 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:30.816882 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39819
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:30.818145 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:30.819458 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:30.832072 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:30.832032 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:30.832094 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:30.832755 1291 server_base.cc:1034] running on GCE node
I20260619 00:59:30.833312 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:30.834177 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:30.835388 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781830770835344 us; error 59 us; skew 500 ppm
I20260619 00:59:30.835721 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:30 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:30.838609 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:30.840057 1291 webserver.cc:460] Webserver started at http://127.0.97.3:33151/ using document root <none> and password file <none>
I20260619 00:59:30.840705 1291 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:30.840847 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:30.841158 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:30.844049 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "4911317c215c4360950005bd917b3336"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "6782136e77511ff0192b29a93c5512e8"
server_key_iv: "12c11bf11e82e9bb9a608178649ec526"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.844866 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4911317c215c4360950005bd917b3336"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "6782136e77511ff0192b29a93c5512e8"
server_key_iv: "12c11bf11e82e9bb9a608178649ec526"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.850020 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:59:30.853574 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.854954 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:30.855134 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "4911317c215c4360950005bd917b3336"
format_stamp: "Formatted at 2026-06-19 00:59:30 on dist-test-slave-3rxd"
server_key: "6782136e77511ff0192b29a93c5512e8"
server_key_iv: "12c11bf11e82e9bb9a608178649ec526"
server_key_version: "encryptionkey@0"
I20260619 00:59:30.855331 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:30.874807 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:30.878562 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:30.878901 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:30.879809 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:30.881075 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:30.881176 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.881315 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:30.881386 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:30.913653 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35695
I20260619 00:59:30.913707 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35695 every 8 connection(s)
I20260619 00:59:30.915015 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:30.915972 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1291
I20260619 00:59:30.916126 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:30.916692 388 external_mini_cluster.cc:1428] Setting key 4da839445d7b35da33010383167f38c2
Jun 19 00:59:30 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:30.934309 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.918097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55661 (local address 127.0.97.62:39819)
0619 00:59:30.918304 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.918310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:30.919803 (+ 1493us) server_negotiation.cc:407] Connection header received
0619 00:59:30.921530 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:30.921534 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:30.921594 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:30.921673 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:30.923116 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.923821 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.925269 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.925638 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.925726 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:30.928997 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:30.929002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:30.929005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:30.929034 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:30.931703 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.932490 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.932494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.932497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.932606 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.932987 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.932990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.932992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.933235 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:30.933336 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:30.933928 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 00:59:30.934075 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 00:59:30.935328 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.918758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39819 (local address 127.0.97.3:55661)
0619 00:59:30.919657 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:30.919706 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:30.921298 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:30.921873 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:30.921881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:30.922355 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:30.922979 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.922997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.923941 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.923945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:30.925153 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.925164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.925809 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.925813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:30.926003 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:30.926663 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:30.926688 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:30.928805 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:30.931821 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:30.931829 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:30.931845 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:30.932381 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:30.932695 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:30.932698 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:30.932700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:30.932892 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:30.933330 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:30.933335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:30.933615 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 00:59:30.933825 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 00:59:30.934187 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":310,"threads_started":1}
I20260619 00:59:30.936605 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39819
I20260619 00:59:30.936890 1419 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:30.937546 1419 heartbeater.cc:511] Master 127.0.97.62:39819 requested a full tablet report, sending...
I20260619 00:59:30.938917 963 ts_manager.cc:194] Registered new tserver with Master: 4911317c215c4360950005bd917b3336 (127.0.97.3:35695)
I20260619 00:59:30.939677 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55661
I20260619 00:59:30.944538 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:59:30.963663 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.950770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.97.62:39819)
0619 00:59:30.950943 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.950947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:30.951455 (+ 508us) server_negotiation.cc:407] Connection header received
0619 00:59:30.951642 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:30.951647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:30.951720 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:30.951854 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:30.953196 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.953897 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.955222 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.955633 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.955764 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:30.957897 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:30.957903 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:30.957908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:30.957955 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:30.961111 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.961723 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.961730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.961734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.961861 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:30.962221 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:30.962227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:30.962231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:30.962531 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:30.962804 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:30.963149 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 00:59:30.963351 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 00:59:30.969733 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:30.976477 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:31.005424 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.988648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.0.97.3:35695)
0619 00:59:30.995799 (+ 7151us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.995812 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:30.996584 (+ 772us) server_negotiation.cc:407] Connection header received
0619 00:59:30.996689 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:30.996697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:30.996979 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:30.997165 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:30.997967 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.999603 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.002328 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.004176 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.004302 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.004556 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.004908 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.005068 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6920,"thread_start_us":6722,"threads_started":1}
I20260619 00:59:31.005494 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.988365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35695 (local address 127.0.0.1:53248)
0619 00:59:30.996168 (+ 7803us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:30.996206 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:30.996389 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:30.997368 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:30.997375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:30.997405 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:30.997659 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.997668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:30.999760 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:30.999765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.002174 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.002189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.004377 (+ 2188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.004383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.004685 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.004720 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:59:31.004879 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 00:59:31.005059 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7563,"mutex_wait_us":87,"thread_start_us":8470,"threads_started":1}
I20260619 00:59:31.007062 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.989179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35337 (local address 127.0.0.1:34324)
0619 00:59:30.997098 (+ 7919us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:30.997115 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:30.997208 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:30.999302 (+ 2094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:30.999307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:30.999329 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:30.999762 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:30.999772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.001844 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.001852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.004683 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.004696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.006237 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.006243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.006393 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.006425 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:59:31.006518 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 00:59:31.006635 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7789,"mutex_wait_us":49,"thread_start_us":6608,"threads_started":1}
I20260619 00:59:31.007231 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.989546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.0.97.1:35337)
0619 00:59:30.998768 (+ 9222us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.998776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:30.998802 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:59:30.998882 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:30.998889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:30.999104 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:30.999973 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:31.000174 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.001691 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.004839 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.006041 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.006143 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.006293 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.006779 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.006945 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9024,"thread_start_us":8875,"threads_started":1}
I20260619 00:59:31.007747 1152 heartbeater.cc:503] Master 127.0.97.62:39819 was elected leader, sending a full tablet report...
I20260619 00:59:31.010835 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.998343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46637 (local address 127.0.0.1:42898)
0619 00:59:31.002333 (+ 3990us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:31.002348 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:31.002468 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:31.003026 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:31.003031 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:31.003059 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:31.003540 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.003553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.005348 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.005353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.007271 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.007284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.010051 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.010057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.010216 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.010245 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:59:31.010338 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 00:59:31.010468 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3818,"mutex_wait_us":26,"thread_start_us":3656,"threads_started":1}
I20260619 00:59:31.010901 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:30.997593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.0.97.2:46637)
0619 00:59:30.998263 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:30.998271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.002441 (+ 4170us) server_negotiation.cc:407] Connection header received
0619 00:59:31.002634 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.002640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.002863 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.003048 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:31.003731 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.005175 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.007448 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.008860 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.008984 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.009172 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.010403 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.010571 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":391,"threads_started":1}
I20260619 00:59:31.010803 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 7637eaf808ab4650ab6b4cba35423469 (DEFAULT_TABLE table=test-table [id=0af14d1cf7c341e2ae74f02fa6b267f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:31.010795 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 7637eaf808ab4650ab6b4cba35423469 (DEFAULT_TABLE table=test-table [id=0af14d1cf7c341e2ae74f02fa6b267f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:31.012109 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7637eaf808ab4650ab6b4cba35423469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:31.012633 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7637eaf808ab4650ab6b4cba35423469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:31.015077 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 7637eaf808ab4650ab6b4cba35423469 (DEFAULT_TABLE table=test-table [id=0af14d1cf7c341e2ae74f02fa6b267f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:31.016355 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7637eaf808ab4650ab6b4cba35423469. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:31.023698 1437 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Bootstrap starting.
I20260619 00:59:31.026378 1438 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Bootstrap starting.
I20260619 00:59:31.028244 1437 tablet_bootstrap.cc:654] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:31.029608 1437 log.cc:824] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:31.029770 1438 tablet_bootstrap.cc:654] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:31.030504 1439 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Bootstrap starting.
I20260619 00:59:31.031141 1438 log.cc:824] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:31.033797 1439 tablet_bootstrap.cc:654] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:31.033931 1438 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: No bootstrap required, opened a new log
I20260619 00:59:31.034186 1438 ts_tablet_manager.cc:1397] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 00:59:31.034812 1437 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: No bootstrap required, opened a new log
I20260619 00:59:31.035072 1437 ts_tablet_manager.cc:1397] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260619 00:59:31.035431 1439 log.cc:824] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:31.038596 1439 tablet_bootstrap.cc:492] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: No bootstrap required, opened a new log
I20260619 00:59:31.038878 1439 ts_tablet_manager.cc:1397] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 00:59:31.042028 1438 raft_consensus.cc:348] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.042389 1438 raft_consensus.cc:374] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:31.042507 1438 raft_consensus.cc:729] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ede99edf3fbb4f448f20c3c53a2cbe15, State: Initialized, Role: FOLLOWER
I20260619 00:59:31.043112 1438 consensus_queue.cc:260] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.044788 1438 ts_tablet_manager.cc:1428] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260619 00:59:31.045742 1437 raft_consensus.cc:348] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.046182 1437 raft_consensus.cc:374] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:31.046280 1437 raft_consensus.cc:729] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4911317c215c4360950005bd917b3336, State: Initialized, Role: FOLLOWER
I20260619 00:59:31.046902 1437 consensus_queue.cc:260] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.048214 1419 heartbeater.cc:503] Master 127.0.97.62:39819 was elected leader, sending a full tablet report...
I20260619 00:59:31.048813 1437 ts_tablet_manager.cc:1428] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 00:59:31.049351 1439 raft_consensus.cc:348] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.049782 1439 raft_consensus.cc:374] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:31.049881 1439 raft_consensus.cc:729] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bcbd5e0a264c7094eaf28167c05e17, State: Initialized, Role: FOLLOWER
I20260619 00:59:31.050449 1439 consensus_queue.cc:260] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.051622 1286 heartbeater.cc:503] Master 127.0.97.62:39819 was elected leader, sending a full tablet report...
I20260619 00:59:31.052527 1439 ts_tablet_manager.cc:1428] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 00:59:31.105218 1445 raft_consensus.cc:482] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:31.105535 1445 raft_consensus.cc:504] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.107278 1445 leader_election.cc:288] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4911317c215c4360950005bd917b3336 (127.0.97.3:35695), ede99edf3fbb4f448f20c3c53a2cbe15 (127.0.97.1:35337)
I20260619 00:59:31.116492 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.107595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56255 (local address 127.0.97.3:35695)
0619 00:59:31.107835 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.107845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.108769 (+ 924us) server_negotiation.cc:407] Connection header received
0619 00:59:31.108890 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.108897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.108970 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.109053 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:31.109714 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.110804 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.113730 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.115411 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.115514 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.115615 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.116115 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.116243 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:59:31.116493 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.107884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46717 (local address 127.0.97.1:35337)
0619 00:59:31.108124 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.108133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.108509 (+ 376us) server_negotiation.cc:407] Connection header received
0619 00:59:31.108742 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.108749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.108834 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.108934 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:31.109614 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.110804 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.113738 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.115369 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.115514 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.115615 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.116115 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.116243 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:59:31.116751 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.107628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35695 (local address 127.0.97.2:56255)
0619 00:59:31.108667 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:31.108681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:31.108782 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:31.109084 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:31.109089 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:31.109122 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:31.109563 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.109575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.110942 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.110946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.113578 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.113592 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.115680 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.115685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.115959 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.115983 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 00:59:31.116076 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 00:59:31.116281 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":208,"thread_start_us":697,"threads_started":1}
I20260619 00:59:31.116884 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.107529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35337 (local address 127.0.97.2:46717)
0619 00:59:31.108388 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:31.108404 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:31.108590 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:31.108959 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:31.108962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:31.108979 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:31.109445 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.109457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.110939 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.110944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.113584 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:31.113592 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.115675 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.115682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:31.115928 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.115981 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 00:59:31.116076 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 00:59:31.116281 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":231,"thread_start_us":520,"threads_started":1}
I20260619 00:59:31.117853 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7637eaf808ab4650ab6b4cba35423469" candidate_uuid: "74bcbd5e0a264c7094eaf28167c05e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4911317c215c4360950005bd917b3336" is_pre_election: true
I20260619 00:59:31.117856 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7637eaf808ab4650ab6b4cba35423469" candidate_uuid: "74bcbd5e0a264c7094eaf28167c05e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" is_pre_election: true
I20260619 00:59:31.118431 1107 raft_consensus.cc:2454] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74bcbd5e0a264c7094eaf28167c05e17 in term 0.
I20260619 00:59:31.118431 1374 raft_consensus.cc:2454] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74bcbd5e0a264c7094eaf28167c05e17 in term 0.
I20260619 00:59:31.119163 1174 leader_election.cc:302] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4911317c215c4360950005bd917b3336, 74bcbd5e0a264c7094eaf28167c05e17, ede99edf3fbb4f448f20c3c53a2cbe15; no voters:
I20260619 00:59:31.119612 1445 raft_consensus.cc:2781] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:31.119781 1445 raft_consensus.cc:482] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:31.119894 1445 raft_consensus.cc:3037] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:31.122092 1445 raft_consensus.cc:504] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.122618 1445 leader_election.cc:288] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [CANDIDATE]: Term 1 election: Requested vote from peers 4911317c215c4360950005bd917b3336 (127.0.97.3:35695), ede99edf3fbb4f448f20c3c53a2cbe15 (127.0.97.1:35337)
I20260619 00:59:31.123064 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7637eaf808ab4650ab6b4cba35423469" candidate_uuid: "74bcbd5e0a264c7094eaf28167c05e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4911317c215c4360950005bd917b3336"
I20260619 00:59:31.123086 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7637eaf808ab4650ab6b4cba35423469" candidate_uuid: "74bcbd5e0a264c7094eaf28167c05e17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15"
I20260619 00:59:31.123257 1374 raft_consensus.cc:3037] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:31.123267 1107 raft_consensus.cc:3037] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:31.125324 1374 raft_consensus.cc:2454] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bcbd5e0a264c7094eaf28167c05e17 in term 1.
I20260619 00:59:31.125367 1107 raft_consensus.cc:2454] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bcbd5e0a264c7094eaf28167c05e17 in term 1.
I20260619 00:59:31.125695 1174 leader_election.cc:302] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4911317c215c4360950005bd917b3336, 74bcbd5e0a264c7094eaf28167c05e17; no voters:
I20260619 00:59:31.125968 1445 raft_consensus.cc:2781] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:31.126845 1445 raft_consensus.cc:686] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [term 1 LEADER]: Becoming Leader. State: Replica: 74bcbd5e0a264c7094eaf28167c05e17, State: Running, Role: LEADER
I20260619 00:59:31.127321 1445 consensus_queue.cc:237] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } }
I20260619 00:59:31.131678 963 catalog_manager.cc:5495] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74bcbd5e0a264c7094eaf28167c05e17 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "74bcbd5e0a264c7094eaf28167c05e17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4911317c215c4360950005bd917b3336" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74bcbd5e0a264c7094eaf28167c05e17" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ede99edf3fbb4f448f20c3c53a2cbe15" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35337 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:31.166857 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.152220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34326 (local address 127.0.97.1:35337)
0619 00:59:31.152444 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.152451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.152989 (+ 538us) server_negotiation.cc:407] Connection header received
0619 00:59:31.153121 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.153126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.153207 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.153290 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.154644 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.155443 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.156591 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.156996 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.157086 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.161155 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.161168 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.161184 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.161216 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.164364 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.164985 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.164991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.164994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.165103 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.165416 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.165419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.165421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.165752 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.166170 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.166316 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.166481 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 00:59:31.167059 1420 tablet.cc:2369] T 7637eaf808ab4650ab6b4cba35423469 P 4911317c215c4360950005bd917b3336: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 00:59:31.195081 1287 tablet.cc:2369] T 7637eaf808ab4650ab6b4cba35423469 P 74bcbd5e0a264c7094eaf28167c05e17: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:31.211566 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.197956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34340 (local address 127.0.97.1:35337)
0619 00:59:31.198164 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.198172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.198591 (+ 419us) server_negotiation.cc:407] Connection header received
0619 00:59:31.198850 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.198855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.198923 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.199031 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.200303 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.201021 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.202257 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.202674 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.202845 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.206336 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.206341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.206344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.206374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.209105 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.209722 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.209734 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.209738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.209849 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.210224 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.210230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.210234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.210482 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.210767 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.211030 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.211237 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 00:59:31.228677 1153 tablet.cc:2369] T 7637eaf808ab4650ab6b4cba35423469 P ede99edf3fbb4f448f20c3c53a2cbe15: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:31.253211 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.240242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.0.97.1:35337)
0619 00:59:31.240494 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.240500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.240699 (+ 199us) server_negotiation.cc:407] Connection header received
0619 00:59:31.240911 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.240916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.240976 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.241079 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.242442 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.243200 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.244421 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.244836 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.244973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.248703 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.248708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.248711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.248742 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.251145 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.251756 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.251759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.251761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.251873 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.252169 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.252172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.252174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.252438 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.252612 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.252806 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.252962 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 00:59:31.255113 1067 tablet_service.cc:2908] Rejecting scan request for tablet 7637eaf808ab4650ab6b4cba35423469: Uninitialized: clean time has not yet been initialized
I20260619 00:59:31.258345 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1024
I20260619 00:59:31.270998 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.259608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.0.97.1:35337)
0619 00:59:31.259765 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.259769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.260117 (+ 348us) server_negotiation.cc:407] Connection header received
0619 00:59:31.260219 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.260222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.260272 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.260342 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.261857 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.262514 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.263780 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.264215 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.264341 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.266357 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.266362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.266366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.266401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.268827 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.269380 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.269383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.269385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.269515 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.269824 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.269827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.269829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.270096 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.270312 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.270494 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.270654 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 00:59:31.271648 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:31.360598 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1024
I20260619 00:59:31.378111 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1158
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:31.392212 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.378941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.0.97.2:46637)
0619 00:59:31.379133 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.379137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.379155 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:59:31.379211 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.379215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.379273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.379357 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.380509 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.381218 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.382448 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.382932 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.383025 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.386841 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.386853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.386869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.386904 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.389627 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.390295 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.390303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.390305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.390433 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.390821 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.390825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.390827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.391166 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.391385 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.391663 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.391919 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:59:31.392855 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:31.482926 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1158
I20260619 00:59:31.502561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1291
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:31.519907 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:31.503321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.0.97.3:35695)
0619 00:59:31.503499 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:31.503504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:31.503521 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:59:31.503646 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:31.503650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:31.503711 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:31.503794 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:31.505026 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.505774 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.507117 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:31.507526 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:31.507661 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:31.513376 (+ 5715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:31.513411 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:31.513437 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:31.513486 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:31.517201 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.518120 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.518130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.518134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.518286 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:31.518613 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:31.518616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:31.518618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:31.519075 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:31.519293 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:31.519507 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 00:59:31.519668 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 00:59:31.520534 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:31.607792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1291
I20260619 00:59:31.623847 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 931
I20260619 00:59:31.625029 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:31.731673 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 931
2026-06-19T00:59:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7537 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[1467](info): set up 2 sockets
Jun 19 00:59:31 dist-test-slave-3rxd krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 00:59:32.814620 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:32 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:32.993961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38955
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:33.314649 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:33.314972 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:33.315133 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:33.322359 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:33.322470 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 00:59:33.322558 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:33.322634 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:33.322710 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:33.322821 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:33.332713 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38955
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:33.334021 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:33.335384 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:33.347307 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:33.347236 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:33.347286 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:33.347992 1480 server_base.cc:1034] running on GCE node
I20260619 00:59:33.348876 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:33.350320 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:33.351548 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781830773351482 us; error 53 us; skew 500 ppm
I20260619 00:59:33.351863 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:33 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:33.354842 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:33.356302 1480 webserver.cc:460] Webserver started at http://127.0.97.62:35557/ using document root <none> and password file <none>
I20260619 00:59:33.356998 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:33.357125 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:33.357450 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:33.360363 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "f36a4d0899764ae7bae15d5edfc26f07"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "7339a478f3578c403fde609fb7aabe90"
server_key_iv: "b7e677cb06a3c87936e0602270de28f7"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.361151 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "f36a4d0899764ae7bae15d5edfc26f07"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "7339a478f3578c403fde609fb7aabe90"
server_key_iv: "b7e677cb06a3c87936e0602270de28f7"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.366120 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:59:33.369546 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:33.371018 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:59:33.371199 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "f36a4d0899764ae7bae15d5edfc26f07"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "7339a478f3578c403fde609fb7aabe90"
server_key_iv: "b7e677cb06a3c87936e0602270de28f7"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.371438 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:33.388790 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:33.392988 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:33.393262 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:33.414696 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38955
I20260619 00:59:33.414702 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38955 every 8 connection(s)
I20260619 00:59:33.416090 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:33.419807 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:33.421610 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1480
I20260619 00:59:33.421831 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:33.422482 388 external_mini_cluster.cc:1428] Setting key 59138e52d97da66a15f44ab59d8094ba
I20260619 00:59:33.432235 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Bootstrap starting.
I20260619 00:59:33.436502 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Neither blocks nor log segments found. Creating new log.
Jun 19 00:59:33 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:33.437757 1557 log.cc:824] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:33.441327 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: No bootstrap required, opened a new log
I20260619 00:59:33.447770 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:33.424458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.97.62:38955)
0619 00:59:33.425251 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:33.425262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:33.425307 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:59:33.426998 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:33.427019 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:33.427382 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:33.427701 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:33.429665 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:33.430988 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:33.432768 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:33.433432 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:33.433564 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:33.438679 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:33.438688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:33.438703 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:33.438834 (+ 131us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:33.442973 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:33.443640 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:33.443650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:33.443658 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:33.443875 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:33.444277 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:33.444283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:33.444287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:33.445047 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:33.445337 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:33.445798 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 00:59:33.446259 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":387,"threads_started":1}
I20260619 00:59:33.451814 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } }
I20260619 00:59:33.452147 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:33.452301 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36a4d0899764ae7bae15d5edfc26f07, State: Initialized, Role: FOLLOWER
I20260619 00:59:33.452970 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } }
I20260619 00:59:33.453275 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:33.453442 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:33.453635 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:33.455948 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } }
I20260619 00:59:33.456488 1557 leader_election.cc:302] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36a4d0899764ae7bae15d5edfc26f07; no voters:
I20260619 00:59:33.457141 1557 leader_election.cc:288] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:33.457274 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:33.458020 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [term 1 LEADER]: Becoming Leader. State: Replica: f36a4d0899764ae7bae15d5edfc26f07, State: Running, Role: LEADER
I20260619 00:59:33.458649 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } }
I20260619 00:59:33.459550 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:33.461735 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36a4d0899764ae7bae15d5edfc26f07. Latest consensus state: current_term: 1 leader_uuid: "f36a4d0899764ae7bae15d5edfc26f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } } }
I20260619 00:59:33.461761 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36a4d0899764ae7bae15d5edfc26f07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36a4d0899764ae7bae15d5edfc26f07" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38955 } } }
I20260619 00:59:33.462195 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:33.462271 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:33.463037 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:33.471786 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:33.483355 1568 catalog_manager.cc:1347] Generated new cluster ID: 1f8852870db043d696f5c1aa8588ae9d
I20260619 00:59:33.483484 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:33.519495 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:33.520751 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:33.534812 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Generated new TSK 0
I20260619 00:59:33.535989 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:33.581871 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38955
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:33.905454 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:33.905740 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:33.905835 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:33.913093 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:33.913203 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:33.913288 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:33.913376 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:33.923188 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:33.924441 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:33.925720 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:33.939488 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:33.939538 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:33.939602 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:33.940506 1583 server_base.cc:1034] running on GCE node
I20260619 00:59:33.941206 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:33.942358 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:33.943614 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781830773943548 us; error 45 us; skew 500 ppm
I20260619 00:59:33.943951 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:33 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:33.950322 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:33.951761 1583 webserver.cc:460] Webserver started at http://127.0.97.1:42359/ using document root <none> and password file <none>
I20260619 00:59:33.952421 1583 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:33.952507 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:33.952772 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:33.956403 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "fdfbe7832fdd43fa91336b7c78b09396"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "d82a3dc8792ce5d63fd3a203a6b0cc9e"
server_key_iv: "ab246712900a02194fe37d8471868168"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.957396 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fdfbe7832fdd43fa91336b7c78b09396"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "d82a3dc8792ce5d63fd3a203a6b0cc9e"
server_key_iv: "ab246712900a02194fe37d8471868168"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.962970 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 00:59:33.966400 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:33.967764 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:33.967958 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "fdfbe7832fdd43fa91336b7c78b09396"
format_stamp: "Formatted at 2026-06-19 00:59:33 on dist-test-slave-3rxd"
server_key: "d82a3dc8792ce5d63fd3a203a6b0cc9e"
server_key_iv: "ab246712900a02194fe37d8471868168"
server_key_version: "encryptionkey@0"
I20260619 00:59:33.968178 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:33.990803 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:33.995260 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:33.995560 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:33.998413 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:33.998522 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:33.998682 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:33.998821 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 00:59:34 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:34.023018 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:33.998949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38395 (local address 127.0.97.62:38955)
0619 00:59:34.001920 (+ 2971us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:34.001925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:34.001943 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:59:34.004445 (+ 2502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:34.004452 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:34.004548 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:34.004697 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:34.006862 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.008170 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.009883 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.010639 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.010795 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.016256 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:34.016263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:34.016269 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:34.016311 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:34.019087 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.020215 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.020219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.020221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.020346 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.020921 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.020924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.020926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.021179 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:34.021283 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:34.022339 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0619 00:59:34.022502 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2871,"thread_start_us":2764,"threads_started":1}
I20260619 00:59:34.024931 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:33.999849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.1:38395)
0619 00:59:34.000953 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:34.001017 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:34.004047 (+ 3030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:34.004984 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:34.005002 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:34.005780 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:34.006622 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.006651 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.008348 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.008354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.009730 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.009742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.010960 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.010966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.011198 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.012059 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:34.012086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:34.015970 (+ 3884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:34.019238 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.019252 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.019281 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.020068 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.020466 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.020474 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.020480 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.020797 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.021303 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:34.021315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:34.021894 (+ 579us) client_negotiation.cc:769] Sending connection context
0619 00:59:34.022264 (+ 370us) client_negotiation.cc:240] Negotiation successful
0619 00:59:34.023082 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":460,"threads_started":1}
I20260619 00:59:34.047932 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.032137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.1:42967)
0619 00:59:34.032834 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:34.032857 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:34.033024 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:34.033426 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:34.033430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:34.033982 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:34.034285 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.034298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.035997 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.036005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.037743 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.037765 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.038920 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.038929 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.039144 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.039883 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:34.039899 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:34.040604 (+ 705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:34.045079 (+ 4475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.045093 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.045098 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.045658 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.046154 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.046157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.046162 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.046280 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.046899 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:34.046904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:34.047146 (+ 242us) client_negotiation.cc:769] Sending connection context
0619 00:59:34.047357 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 00:59:34.047617 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":391,"threads_started":1}
I20260619 00:59:34.048287 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.032180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42967 (local address 127.0.97.62:38955)
0619 00:59:34.032414 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:34.032423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:34.032949 (+ 526us) server_negotiation.cc:407] Connection header received
0619 00:59:34.033228 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:34.033232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:34.033297 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:34.033401 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:34.034470 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.035840 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.037934 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.038694 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.038875 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.040807 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:34.040817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:34.040825 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:34.040875 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:34.044881 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.045817 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.045826 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.045832 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.046018 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.046433 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.046440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.046446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.046705 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:34.046902 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:34.047509 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 00:59:34.047794 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:59:34.058763 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40569
I20260619 00:59:34.058789 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40569 every 8 connection(s)
I20260619 00:59:34.060097 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:34.062098 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38955
I20260619 00:59:34.062279 1719 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:34.062934 1719 heartbeater.cc:511] Master 127.0.97.62:38955 requested a full tablet report, sending...
I20260619 00:59:34.064759 1512 ts_manager.cc:194] Registered new tserver with Master: fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569)
I20260619 00:59:34.066120 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38395
I20260619 00:59:34.069942 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1583
I20260619 00:59:34.070165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:34.070807 388 external_mini_cluster.cc:1428] Setting key f20017e25306cffc15f988298c9ae6b4
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:34.108183 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38955
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:34.429157 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:34.429435 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:34.429612 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:34.436914 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:34.437054 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:34.437214 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:34.437307 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:34.447268 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:34.448604 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:34.449954 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:34.463500 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:34.463527 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:34.463691 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:34.464305 1723 server_base.cc:1034] running on GCE node
I20260619 00:59:34.464977 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:34.465960 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:34.467170 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781830774467098 us; error 57 us; skew 500 ppm
I20260619 00:59:34.467433 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:34 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:34.470420 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:34.471999 1723 webserver.cc:460] Webserver started at http://127.0.97.2:43105/ using document root <none> and password file <none>
I20260619 00:59:34.472787 1723 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:34.472873 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:34.473132 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:34.476274 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4dc474a5e8c4319a1117c1b4c574294"
format_stamp: "Formatted at 2026-06-19 00:59:34 on dist-test-slave-3rxd"
server_key: "2b1815c40cc49e62e2ac25fabdda455f"
server_key_iv: "5df3eed89d4936ff9e4370f638e84d14"
server_key_version: "encryptionkey@0"
I20260619 00:59:34.477002 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4dc474a5e8c4319a1117c1b4c574294"
format_stamp: "Formatted at 2026-06-19 00:59:34 on dist-test-slave-3rxd"
server_key: "2b1815c40cc49e62e2ac25fabdda455f"
server_key_iv: "5df3eed89d4936ff9e4370f638e84d14"
server_key_version: "encryptionkey@0"
I20260619 00:59:34.481968 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260619 00:59:34.485786 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:34.487213 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 00:59:34.487385 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "f4dc474a5e8c4319a1117c1b4c574294"
format_stamp: "Formatted at 2026-06-19 00:59:34 on dist-test-slave-3rxd"
server_key: "2b1815c40cc49e62e2ac25fabdda455f"
server_key_iv: "5df3eed89d4936ff9e4370f638e84d14"
server_key_version: "encryptionkey@0"
I20260619 00:59:34.487599 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:34.507105 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:34.511437 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:34.511724 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:34.514791 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:34.514899 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 00:59:34.515044 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:34.515113 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:59:34 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:34.540190 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.515558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34475 (local address 127.0.97.62:38955)
0619 00:59:34.515734 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:34.515740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:34.517555 (+ 1815us) server_negotiation.cc:407] Connection header received
0619 00:59:34.519817 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:34.519824 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:34.519913 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:34.519987 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:34.522784 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.524137 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.526661 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.527371 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.527533 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.532610 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:34.532615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:34.532619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:34.532677 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:34.536190 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.537477 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.537481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.537484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.537607 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.538195 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.538199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.538201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.538480 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:34.538585 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:34.539771 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0619 00:59:34.539957 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 00:59:34.542532 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.516496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.2:34475)
0619 00:59:34.517365 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:34.517420 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:34.519435 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:34.520560 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:34.520574 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:34.521478 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:34.522417 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.522471 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.524375 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.524394 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.526483 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.526495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.527948 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.527959 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.528305 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.529088 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:34.529112 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:34.532274 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:34.536352 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.536366 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.536392 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.537315 (+ 923us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.537742 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.537750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.537756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.538073 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.538621 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:34.538633 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:34.539256 (+ 623us) client_negotiation.cc:769] Sending connection context
0619 00:59:34.539683 (+ 427us) client_negotiation.cc:240] Negotiation successful
0619 00:59:34.540420 (+ 737us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":399,"threads_started":1}
I20260619 00:59:34.564500 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.549094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.2:39713)
0619 00:59:34.549748 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:34.549767 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:34.549948 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:34.550269 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:34.550273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:34.550780 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:34.551198 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.551216 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.552942 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.552946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.553926 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:34.553936 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.555248 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.555253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:34.555380 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.556123 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:34.556145 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:34.557585 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:34.561084 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.561093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.561100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.561870 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.562221 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:34.562229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:34.562235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:34.562423 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:34.563205 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:34.563209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:34.563437 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 00:59:34.563627 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 00:59:34.563970 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":383,"threads_started":1}
I20260619 00:59:34.565598 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:34.549009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39713 (local address 127.0.97.62:38955)
0619 00:59:34.549167 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:34.549171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:34.549857 (+ 686us) server_negotiation.cc:407] Connection header received
0619 00:59:34.550082 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:34.550086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:34.550146 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:34.550247 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:34.551384 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.552772 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.554062 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:34.555020 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:34.555202 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:34.557799 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:34.557808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:34.557815 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:34.557856 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:34.560949 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.561995 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.561999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.562002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.562110 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:34.562542 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:34.562545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:34.562547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:34.563015 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:34.563157 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:34.565161 (+ 2004us) server_negotiation.cc:299] Negotiation successful
0619 00:59:34.565395 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 00:59:34.577450 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45367
I20260619 00:59:34.577497 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45367 every 8 connection(s)
I20260619 00:59:34.578698 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:34.580861 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38955
I20260619 00:59:34.581120 1858 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:34.581770 1858 heartbeater.cc:511] Master 127.0.97.62:38955 requested a full tablet report, sending...
I20260619 00:59:34.583128 1512 ts_manager.cc:194] Registered new tserver with Master: f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367)
I20260619 00:59:34.583966 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34475
I20260619 00:59:34.585902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1723
I20260619 00:59:34.586107 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:34.586606 388 external_mini_cluster.cc:1428] Setting key 01323fee26eeb448c8860fd097f06f75
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:34.626221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38955
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:34.955641 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:34.955986 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:34.956164 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:34.965070 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:34.965236 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:34.965412 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:34.965562 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:34.976488 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:34.977835 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:34.979399 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:34.992432 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:34.992511 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:34.992691 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:34.996347 1862 server_base.cc:1034] running on GCE node
I20260619 00:59:34.996989 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:34.997931 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:34.999174 1862 hybrid_clock.cc:648] HybridClock initialized: now 1781830774999117 us; error 36 us; skew 500 ppm
I20260619 00:59:34.999473 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:34 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:35.002506 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:35.004055 1862 webserver.cc:460] Webserver started at http://127.0.97.3:38515/ using document root <none> and password file <none>
I20260619 00:59:35.004781 1862 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:35.004909 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:35.005230 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:35.008220 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "d08fb92c32a94271b4682ab9e34c8bf9"
format_stamp: "Formatted at 2026-06-19 00:59:35 on dist-test-slave-3rxd"
server_key: "49077bca4ffd4a174d576e0d6aea9a4b"
server_key_iv: "093603d88a9ca66e861333b6ed1bc6de"
server_key_version: "encryptionkey@0"
I20260619 00:59:35.008985 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d08fb92c32a94271b4682ab9e34c8bf9"
format_stamp: "Formatted at 2026-06-19 00:59:35 on dist-test-slave-3rxd"
server_key: "49077bca4ffd4a174d576e0d6aea9a4b"
server_key_iv: "093603d88a9ca66e861333b6ed1bc6de"
server_key_version: "encryptionkey@0"
I20260619 00:59:35.013969 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 00:59:35.017338 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:35.018649 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:59:35.018848 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "d08fb92c32a94271b4682ab9e34c8bf9"
format_stamp: "Formatted at 2026-06-19 00:59:35 on dist-test-slave-3rxd"
server_key: "49077bca4ffd4a174d576e0d6aea9a4b"
server_key_iv: "093603d88a9ca66e861333b6ed1bc6de"
server_key_version: "encryptionkey@0"
I20260619 00:59:35.019070 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:35.032910 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:35.036751 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:35.037026 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:35.039685 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:35.039799 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:35.039979 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:35.040078 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 00:59:35 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:35.064678 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.040208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34863 (local address 127.0.97.62:38955)
0619 00:59:35.040450 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.040460 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.041847 (+ 1387us) server_negotiation.cc:407] Connection header received
0619 00:59:35.045072 (+ 3225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.045079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.045170 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.045310 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:35.048092 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.049393 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.051401 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.051984 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.052126 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.057840 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:35.057846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:35.057849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:35.057886 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:35.061052 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.061839 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.061852 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.061855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.061982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.062541 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.062554 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.062556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.063007 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:35.063163 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.064035 (+ 872us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.064264 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:59:35.065891 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.040753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.3:34863)
0619 00:59:35.041651 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.041714 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.044580 (+ 2866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.045867 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.045892 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.046819 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:35.047831 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.047857 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.049597 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.049605 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.051127 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.051168 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.052241 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.052249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.052637 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.053709 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:35.053751 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:35.057570 (+ 3819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:35.061220 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.061230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.061246 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.061692 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.062216 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.062220 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.062222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.062404 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.063224 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:35.063229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:35.063542 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.063886 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.064424 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":363,"threads_started":1}
I20260619 00:59:35.068697 1719 heartbeater.cc:503] Master 127.0.97.62:38955 was elected leader, sending a full tablet report...
I20260619 00:59:35.089084 1932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.072181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.97.3:51783)
0619 00:59:35.073076 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.073102 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.073263 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.073640 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.073647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.074401 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:35.074962 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.075054 (+ 92us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.076465 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.076474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.078626 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.078644 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.079644 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.079653 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.079955 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.080790 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:35.080807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:35.082098 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:35.086099 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.086107 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.086111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.086693 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.087245 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.087251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.087255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.087497 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.088251 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:35.088257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:35.088428 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.088534 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.088765 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":480,"threads_started":1}
I20260619 00:59:35.089537 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.072175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51783 (local address 127.0.97.62:38955)
0619 00:59:35.072389 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.072399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.073201 (+ 802us) server_negotiation.cc:407] Connection header received
0619 00:59:35.073394 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.073398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.073483 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.073586 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:35.075215 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.076331 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.078812 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.079412 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.079555 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.082288 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:35.082294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:35.082299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:35.082353 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:35.085951 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.086947 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.086954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.086960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.087116 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.087650 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.087659 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.087665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.088112 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:35.088286 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.088724 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.089030 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 00:59:35.102417 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35257
I20260619 00:59:35.102451 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35257 every 8 connection(s)
I20260619 00:59:35.103698 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:35.104688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 1862
I20260619 00:59:35.104913 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:35.105406 388 external_mini_cluster.cc:1428] Setting key 632d51e065d7603d677d442740c0b061
I20260619 00:59:35.106063 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38955
I20260619 00:59:35.106289 1997 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:35.107162 1997 heartbeater.cc:511] Master 127.0.97.62:38955 requested a full tablet report, sending...
I20260619 00:59:35.108485 1511 ts_manager.cc:194] Registered new tserver with Master: d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257)
I20260619 00:59:35.109318 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34863
I20260619 00:59:35.120850 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:59:35 dist-test-slave-3rxd krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:59:35 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:35.165526 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.152360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.0.97.62:38955)
0619 00:59:35.152597 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.152605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.152782 (+ 177us) server_negotiation.cc:407] Connection header received
0619 00:59:35.153011 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.153016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.153078 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.153179 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:35.154553 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.155324 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.156602 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.157031 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.157176 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.161030 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:35.161035 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:35.161038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:35.161071 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:35.163428 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.164002 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.164005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.164008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.164130 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.164444 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.164447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.164449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.164697 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:35.164867 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.165092 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.165245 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 00:59:35.169855 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:35.177191 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:35.211202 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.188866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35257 (local address 127.0.0.1:33128)
0619 00:59:35.200941 (+ 12075us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.200990 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.201236 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.203107 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.203113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.203137 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.203751 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.203765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.205658 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.205664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.208499 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.208514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.210353 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.210362 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.210541 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210575 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.210652 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.210820 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11605,"mutex_wait_us":211,"thread_start_us":13766,"threads_started":1}
I20260619 00:59:35.211285 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.189040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.3:35257)
0619 00:59:35.200120 (+ 11080us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.200132 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.201249 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 00:59:35.201533 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.201541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.201856 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.202107 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.203967 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.205518 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.208689 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.210152 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.210284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210503 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.210768 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.210942 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10875,"thread_start_us":10668,"threads_started":1}
I20260619 00:59:35.211416 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.189803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.97.1:40569)
0619 00:59:35.201813 (+ 12010us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.201824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.203001 (+ 1177us) server_negotiation.cc:407] Connection header received
0619 00:59:35.203235 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.203242 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.203451 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.203619 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.204360 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.205995 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.208191 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.209774 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.209931 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210157 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.210925 (+ 768us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.211098 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11805,"thread_start_us":11650,"threads_started":1}
I20260619 00:59:35.211800 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.189683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.0.97.2:45367)
0619 00:59:35.199271 (+ 9588us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.199285 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.202385 (+ 3100us) server_negotiation.cc:407] Connection header received
0619 00:59:35.202556 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.202567 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.202824 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.203046 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.203785 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.205401 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.208741 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.210152 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.210286 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210473 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.211314 (+ 841us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.211491 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9369,"thread_start_us":9625,"threads_started":1}
I20260619 00:59:35.211992 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.189637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:40284)
0619 00:59:35.202890 (+ 13253us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.202904 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.203051 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.203860 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.203867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.203892 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.204189 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.204195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.206124 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.206130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.208041 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.208053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.209957 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.209963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.210163 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210240 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.210392 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.210604 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13164,"mutex_wait_us":51,"thread_start_us":12379,"threads_started":1}
I20260619 00:59:35.212340 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.189407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45367 (local address 127.0.0.1:37998)
0619 00:59:35.202266 (+ 12859us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.202288 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.202412 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.203084 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.203094 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.203124 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.203602 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.203615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.205562 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.205570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.208575 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.208589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.210621 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.210627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.210847 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.210879 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.210978 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.211115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12740,"mutex_wait_us":217,"thread_start_us":10913,"threads_started":1}
I20260619 00:59:35.215418 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 07c04aa3d13f430997caa974489bbe66 (DEFAULT_TABLE table=test-table [id=010f767b6ec2488f938050e7f96f8f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.215534 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 07c04aa3d13f430997caa974489bbe66 (DEFAULT_TABLE table=test-table [id=010f767b6ec2488f938050e7f96f8f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.216703 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c04aa3d13f430997caa974489bbe66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.216837 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c04aa3d13f430997caa974489bbe66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.217047 1649 tablet_service.cc:1467] Processing CreateTablet for tablet 07c04aa3d13f430997caa974489bbe66 (DEFAULT_TABLE table=test-table [id=010f767b6ec2488f938050e7f96f8f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.218256 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c04aa3d13f430997caa974489bbe66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.228804 2016 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Bootstrap starting.
I20260619 00:59:35.231014 2017 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Bootstrap starting.
I20260619 00:59:35.233771 2016 tablet_bootstrap.cc:654] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.234282 2017 tablet_bootstrap.cc:654] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.235150 2016 log.cc:824] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:35.235469 2017 log.cc:824] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:35.235991 2018 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Bootstrap starting.
I20260619 00:59:35.239815 2018 tablet_bootstrap.cc:654] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.239868 2017 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: No bootstrap required, opened a new log
I20260619 00:59:35.240208 2017 ts_tablet_manager.cc:1397] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260619 00:59:35.240904 2018 log.cc:824] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:35.241142 2016 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: No bootstrap required, opened a new log
I20260619 00:59:35.241394 2016 ts_tablet_manager.cc:1397] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260619 00:59:35.244729 2018 tablet_bootstrap.cc:492] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: No bootstrap required, opened a new log
I20260619 00:59:35.245031 2018 ts_tablet_manager.cc:1397] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 00:59:35.251411 2017 raft_consensus.cc:348] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.252287 2017 raft_consensus.cc:374] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.252424 2017 raft_consensus.cc:729] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dc474a5e8c4319a1117c1b4c574294, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.252655 2018 raft_consensus.cc:348] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.253064 2018 raft_consensus.cc:374] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.253159 2018 raft_consensus.cc:729] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbe7832fdd43fa91336b7c78b09396, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.253067 2017 consensus_queue.cc:260] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.253041 2016 raft_consensus.cc:348] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.253453 2016 raft_consensus.cc:374] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.253626 2016 raft_consensus.cc:729] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08fb92c32a94271b4682ab9e34c8bf9, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.253712 2018 consensus_queue.cc:260] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.254338 2016 consensus_queue.cc:260] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.255529 2018 ts_tablet_manager.cc:1428] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260619 00:59:35.259496 1858 heartbeater.cc:503] Master 127.0.97.62:38955 was elected leader, sending a full tablet report...
I20260619 00:59:35.260164 2017 ts_tablet_manager.cc:1428] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Time spent starting tablet: real 0.020s user 0.015s sys 0.004s
I20260619 00:59:35.263479 1997 heartbeater.cc:503] Master 127.0.97.62:38955 was elected leader, sending a full tablet report...
I20260619 00:59:35.264197 2016 ts_tablet_manager.cc:1428] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent starting tablet: real 0.023s user 0.018s sys 0.000s
I20260619 00:59:35.291004 2024 raft_consensus.cc:482] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:35.291381 2024 raft_consensus.cc:504] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.293249 2024 leader_election.cc:288] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367), fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569)
I20260619 00:59:35.300417 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.293428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45367 (local address 127.0.97.3:58609)
0619 00:59:35.293698 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.293725 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.293896 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.294360 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.294366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.294399 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.294651 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.294665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.296379 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.296382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.298215 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.298229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.299665 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.299672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.299882 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.299919 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.300031 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.300172 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 00:59:35.300485 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.293544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58609 (local address 127.0.97.2:45367)
0619 00:59:35.293739 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.293747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.293792 (+ 45us) server_negotiation.cc:407] Connection header received
0619 00:59:35.294077 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.294086 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.294184 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.294310 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.294820 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.295829 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.298426 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.299488 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.299637 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.299752 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.300062 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.300200 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 00:59:35.300972 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.293820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.97.3:55501)
0619 00:59:35.294611 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.294628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.294785 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.295107 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.295110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.295127 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.295387 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.295393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.296941 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.296947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.298786 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.298796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.300170 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.300176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.300348 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.300382 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.300485 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.300619 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":406,"threads_started":1}
I20260619 00:59:35.301187 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.293832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55501 (local address 127.0.97.1:40569)
0619 00:59:35.294051 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.294061 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.294756 (+ 695us) server_negotiation.cc:407] Connection header received
0619 00:59:35.294917 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.294922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.294996 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.295093 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.295535 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.296807 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.298922 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.299953 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.300079 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.300169 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.300810 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.300943 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:59:35.301523 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c04aa3d13f430997caa974489bbe66" candidate_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc474a5e8c4319a1117c1b4c574294" is_pre_election: true
I20260619 00:59:35.302040 1813 raft_consensus.cc:2454] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08fb92c32a94271b4682ab9e34c8bf9 in term 0.
I20260619 00:59:35.302644 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c04aa3d13f430997caa974489bbe66" candidate_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbe7832fdd43fa91336b7c78b09396" is_pre_election: true
I20260619 00:59:35.302793 1881 leader_election.cc:302] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d08fb92c32a94271b4682ab9e34c8bf9, f4dc474a5e8c4319a1117c1b4c574294; no voters:
I20260619 00:59:35.303180 1674 raft_consensus.cc:2454] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d08fb92c32a94271b4682ab9e34c8bf9 in term 0.
I20260619 00:59:35.303200 2024 raft_consensus.cc:2781] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:35.303357 2024 raft_consensus.cc:482] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:35.303488 2024 raft_consensus.cc:3037] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.305672 2024 raft_consensus.cc:504] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
I20260619 00:59:35.306322 2024 leader_election.cc:288] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [CANDIDATE]: Term 1 election: Requested vote from peers f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367), fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569)
I20260619 00:59:35.306762 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c04aa3d13f430997caa974489bbe66" candidate_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc474a5e8c4319a1117c1b4c574294"
I20260619 00:59:35.306851 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c04aa3d13f430997caa974489bbe66" candidate_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbe7832fdd43fa91336b7c78b09396"
I20260619 00:59:35.306972 1813 raft_consensus.cc:3037] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.307056 1674 raft_consensus.cc:3037] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.309269 1674 raft_consensus.cc:2454] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08fb92c32a94271b4682ab9e34c8bf9 in term 1.
I20260619 00:59:35.309276 1813 raft_consensus.cc:2454] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d08fb92c32a94271b4682ab9e34c8bf9 in term 1.
I20260619 00:59:35.309756 1879 leader_election.cc:302] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: d08fb92c32a94271b4682ab9e34c8bf9, f4dc474a5e8c4319a1117c1b4c574294, fdfbe7832fdd43fa91336b7c78b09396; no voters:
I20260619 00:59:35.310109 2024 raft_consensus.cc:2781] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:35.310549 2024 raft_consensus.cc:686] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 LEADER]: Becoming Leader. State: Replica: d08fb92c32a94271b4682ab9e34c8bf9, State: Running, Role: LEADER
I20260619 00:59:35.311094 2024 consensus_queue.cc:237] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } }
W20260619 00:59:35.312450 1720 tablet.cc:2369] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:59:35.315667 1511 catalog_manager.cc:5495] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } health_report { overall_health: UNKNOWN } } }
W20260619 00:59:35.331030 1859 tablet.cc:2369] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:59:35.340850 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.335722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.0.97.3:35257)
0619 00:59:35.335951 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.335959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.336124 (+ 165us) server_negotiation.cc:407] Connection header received
0619 00:59:35.336416 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.336420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.336493 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.336606 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:35.336906 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.337585 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.338873 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.339272 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.339363 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.339808 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.340288 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.340396 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.340481 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.340541 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 00:59:35.349177 1674 raft_consensus.cc:1261] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Refusing update from remote peer d08fb92c32a94271b4682ab9e34c8bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.349179 1813 raft_consensus.cc:1261] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Refusing update from remote peer d08fb92c32a94271b4682ab9e34c8bf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.350207 2024 consensus_queue.cc:1035] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:35.350524 2026 consensus_queue.cc:1035] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:59:35.355158 1998 tablet.cc:2369] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:59:35.362846 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298378855821000704 to 7298378855678164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298378855821000704}]
I20260619 00:59:35.365454 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298378855821000704 to 7298378855678164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298378855821000704 or (T in {7298378855821000704})}]
I20260619 00:59:35.367233 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298378855821000704 to 7298378855678164992. Current Snapshot: MvccSnapshot[applied={T|T < 7298378855821000704}]
I20260619 00:59:35.383028 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:35.383288 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:35.386929 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Sending DeleteTablet for 3 replicas of tablet 07c04aa3d13f430997caa974489bbe66
I20260619 00:59:35.388068 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c04aa3d13f430997caa974489bbe66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33128
I20260619 00:59:35.388159 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c04aa3d13f430997caa974489bbe66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37998
I20260619 00:59:35.388340 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c04aa3d13f430997caa974489bbe66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40284
I20260619 00:59:35.390808 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:35.392223 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:35.401441 2040 tablet_replica.cc:330] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: stopping tablet replica
I20260619 00:59:35.401973 2040 raft_consensus.cc:2229] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:35.402506 2040 raft_consensus.cc:2258] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:35.404825 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 (DEFAULT_TABLE table=test-table [id=f40b16979fa54d54b101d0718055d4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.404827 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 (DEFAULT_TABLE table=test-table [id=f40b16979fa54d54b101d0718055d4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.405426 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0e8207ef94cba89ac4aa83bd0b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.405441 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0e8207ef94cba89ac4aa83bd0b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.408564 2039 tablet_replica.cc:330] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: stopping tablet replica
I20260619 00:59:35.409899 2039 raft_consensus.cc:2229] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:35.410684 2038 tablet_replica.cc:330] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: stopping tablet replica
I20260619 00:59:35.410900 2039 raft_consensus.cc:2258] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:35.411222 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 (DEFAULT_TABLE table=test-table [id=f40b16979fa54d54b101d0718055d4ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:35.411240 2038 raft_consensus.cc:2229] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:35.411705 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39c0e8207ef94cba89ac4aa83bd0b179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.413084 2016 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Bootstrap starting.
I20260619 00:59:35.413339 2038 raft_consensus.cc:2258] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:35.414109 2039 ts_tablet_manager.cc:1905] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:35.414919 2016 tablet_bootstrap.cc:654] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.417143 2040 ts_tablet_manager.cc:1905] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:35.425539 2039 ts_tablet_manager.cc:1918] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:59:35.425884 2039 log.cc:1196] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/07c04aa3d13f430997caa974489bbe66
I20260619 00:59:35.426537 2039 ts_tablet_manager.cc:1939] T 07c04aa3d13f430997caa974489bbe66 P f4dc474a5e8c4319a1117c1b4c574294: Deleting consensus metadata
I20260619 00:59:35.429061 2018 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Bootstrap starting.
I20260619 00:59:35.429778 2017 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Bootstrap starting.
I20260619 00:59:35.429898 1499 catalog_manager.cc:4841] TS f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367): tablet 07c04aa3d13f430997caa974489bbe66 (table test-table [id=010f767b6ec2488f938050e7f96f8f93]) successfully deleted
I20260619 00:59:35.431419 2017 tablet_bootstrap.cc:654] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.431629 2040 ts_tablet_manager.cc:1918] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:59:35.431792 2040 log.cc:1196] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/07c04aa3d13f430997caa974489bbe66
I20260619 00:59:35.431856 2018 tablet_bootstrap.cc:654] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.432768 2040 ts_tablet_manager.cc:1939] T 07c04aa3d13f430997caa974489bbe66 P fdfbe7832fdd43fa91336b7c78b09396: Deleting consensus metadata
I20260619 00:59:35.437620 2016 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: No bootstrap required, opened a new log
I20260619 00:59:35.437932 2016 ts_tablet_manager.cc:1397] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.009s
I20260619 00:59:35.439249 2016 raft_consensus.cc:348] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.439705 2016 raft_consensus.cc:374] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.439874 2016 raft_consensus.cc:729] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08fb92c32a94271b4682ab9e34c8bf9, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.440426 2017 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: No bootstrap required, opened a new log
I20260619 00:59:35.440322 2016 consensus_queue.cc:260] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.440783 2017 ts_tablet_manager.cc:1397] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260619 00:59:35.441577 2017 raft_consensus.cc:348] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.441859 2017 raft_consensus.cc:374] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.442003 2017 raft_consensus.cc:729] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dc474a5e8c4319a1117c1b4c574294, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.442349 2017 consensus_queue.cc:260] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.444159 2016 ts_tablet_manager.cc:1428] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260619 00:59:35.444360 2038 ts_tablet_manager.cc:1905] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:35.445225 2017 ts_tablet_manager.cc:1428] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260619 00:59:35.448305 1497 catalog_manager.cc:4841] TS fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569): tablet 07c04aa3d13f430997caa974489bbe66 (table test-table [id=010f767b6ec2488f938050e7f96f8f93]) successfully deleted
I20260619 00:59:35.454880 2038 ts_tablet_manager.cc:1918] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 00:59:35.455163 2038 log.cc:1196] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/07c04aa3d13f430997caa974489bbe66
I20260619 00:59:35.455842 2038 ts_tablet_manager.cc:1939] T 07c04aa3d13f430997caa974489bbe66 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting consensus metadata
I20260619 00:59:35.458132 1496 catalog_manager.cc:4841] TS d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257): tablet 07c04aa3d13f430997caa974489bbe66 (table test-table [id=010f767b6ec2488f938050e7f96f8f93]) successfully deleted
I20260619 00:59:35.459933 2018 tablet_bootstrap.cc:492] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: No bootstrap required, opened a new log
I20260619 00:59:35.460103 2018 ts_tablet_manager.cc:1397] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.017s
I20260619 00:59:35.460932 2018 raft_consensus.cc:348] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.461179 2018 raft_consensus.cc:374] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.461272 2018 raft_consensus.cc:729] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbe7832fdd43fa91336b7c78b09396, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.461596 2018 consensus_queue.cc:260] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.462927 2018 ts_tablet_manager.cc:1428] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 00:59:35.589956 2022 raft_consensus.cc:482] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:35.590209 2022 raft_consensus.cc:504] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.592701 2022 leader_election.cc:288] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569), d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257)
I20260619 00:59:35.608670 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.593108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36803 (local address 127.0.97.1:40569)
0619 00:59:35.593379 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.593385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.601782 (+ 8397us) server_negotiation.cc:407] Connection header received
0619 00:59:35.602417 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.602424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.602514 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.602578 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.603444 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.604185 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.606568 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.607533 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.607660 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.607771 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.608133 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.608283 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 00:59:35.608878 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.593052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.97.2:36803)
0619 00:59:35.601613 (+ 8561us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.601639 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.601928 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.602795 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.602799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.602824 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.603276 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.603288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.604321 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.604324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.606412 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.606425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.607747 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.607753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.607949 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.607984 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.608084 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.608385 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8352,"mutex_wait_us":4,"thread_start_us":8535,"threads_started":1}
I20260619 00:59:35.609673 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0e8207ef94cba89ac4aa83bd0b179" candidate_uuid: "f4dc474a5e8c4319a1117c1b4c574294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbe7832fdd43fa91336b7c78b09396" is_pre_election: true
I20260619 00:59:35.609835 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.593515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35257 (local address 127.0.97.2:57231)
0619 00:59:35.601873 (+ 8358us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.601903 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.602039 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.602590 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.602597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.602623 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.603052 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.603065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.604613 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.604618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.607173 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.607187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.609064 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.609073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.609238 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.609296 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.609381 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.609507 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8219,"thread_start_us":7696,"threads_started":1}
I20260619 00:59:35.609899 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.594071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57231 (local address 127.0.97.3:35257)
0619 00:59:35.594238 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.594244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.602026 (+ 7782us) server_negotiation.cc:407] Connection header received
0619 00:59:35.602163 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.602168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.602246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.602348 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.603270 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.604431 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.607345 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.608675 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.608814 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.608920 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.609446 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.609600 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 00:59:35.609934 1674 raft_consensus.cc:2454] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4dc474a5e8c4319a1117c1b4c574294 in term 0.
I20260619 00:59:35.610798 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0e8207ef94cba89ac4aa83bd0b179" candidate_uuid: "f4dc474a5e8c4319a1117c1b4c574294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" is_pre_election: true
I20260619 00:59:35.611137 1952 raft_consensus.cc:2454] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4dc474a5e8c4319a1117c1b4c574294 in term 0.
I20260619 00:59:35.611260 1740 leader_election.cc:302] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f4dc474a5e8c4319a1117c1b4c574294, fdfbe7832fdd43fa91336b7c78b09396; no voters:
I20260619 00:59:35.611703 2022 raft_consensus.cc:2781] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:35.611825 2022 raft_consensus.cc:482] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:35.611923 2022 raft_consensus.cc:3037] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.614040 2022 raft_consensus.cc:504] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.614703 2022 leader_election.cc:288] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [CANDIDATE]: Term 1 election: Requested vote from peers fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569), d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257)
I20260619 00:59:35.615137 1674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0e8207ef94cba89ac4aa83bd0b179" candidate_uuid: "f4dc474a5e8c4319a1117c1b4c574294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdfbe7832fdd43fa91336b7c78b09396"
I20260619 00:59:35.615351 1674 raft_consensus.cc:3037] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.615398 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39c0e8207ef94cba89ac4aa83bd0b179" candidate_uuid: "f4dc474a5e8c4319a1117c1b4c574294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08fb92c32a94271b4682ab9e34c8bf9"
I20260619 00:59:35.615661 1952 raft_consensus.cc:3037] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.618249 1952 raft_consensus.cc:2454] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4dc474a5e8c4319a1117c1b4c574294 in term 1.
I20260619 00:59:35.618551 1674 raft_consensus.cc:2454] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4dc474a5e8c4319a1117c1b4c574294 in term 1.
I20260619 00:59:35.618780 1739 leader_election.cc:302] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d08fb92c32a94271b4682ab9e34c8bf9, f4dc474a5e8c4319a1117c1b4c574294; no voters:
I20260619 00:59:35.619123 2022 raft_consensus.cc:2781] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:35.619673 2022 raft_consensus.cc:686] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 LEADER]: Becoming Leader. State: Replica: f4dc474a5e8c4319a1117c1b4c574294, State: Running, Role: LEADER
I20260619 00:59:35.620074 2022 consensus_queue.cc:237] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.623752 1511 catalog_manager.cc:5495] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f4dc474a5e8c4319a1117c1b4c574294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:59:35 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:35.697856 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.680934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38955 (local address 127.0.0.1:44856)
0619 00:59:35.681659 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.681681 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.681843 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.682317 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.682322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.683273 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:35.683588 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.683600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.684684 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.684690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.686259 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.686277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.687166 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.687177 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.687757 (+ 580us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.688517 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:35.688541 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:35.691231 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:35.695054 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.695064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.695071 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.695504 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.695956 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:35.695960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:35.695962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:35.696124 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:35.696786 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:35.696792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:35.697179 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.697346 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.697555 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260619 00:59:35.698264 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.681006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44856 (local address 127.0.97.62:38955)
0619 00:59:35.682038 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.682042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.682061 (+ 19us) server_negotiation.cc:407] Connection header received
0619 00:59:35.682104 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.682107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.682159 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.682249 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:35.683764 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.684548 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.686425 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.686967 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.687064 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.691442 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:35.691449 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:35.691455 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:35.691499 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:35.694873 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.695630 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.695635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.695638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.695808 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:35.696246 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:35.696250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:35.696254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:35.696640 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:35.696805 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.697508 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.697766 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":809,"threads_started":1}
I20260619 00:59:35.703747 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 00:59:35.704744 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:35.712289 1648 tablet_service.cc:1467] Processing CreateTablet for tablet ed27ff0232364ada8bedd6d7b1975359 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65883783f3145aeb75b701be4c071da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:35.712852 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed27ff0232364ada8bedd6d7b1975359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.712954 1788 tablet_service.cc:1467] Processing CreateTablet for tablet ed27ff0232364ada8bedd6d7b1975359 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65883783f3145aeb75b701be4c071da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:35.713474 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed27ff0232364ada8bedd6d7b1975359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.717243 1926 tablet_service.cc:1467] Processing CreateTablet for tablet ed27ff0232364ada8bedd6d7b1975359 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65883783f3145aeb75b701be4c071da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:35.717723 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed27ff0232364ada8bedd6d7b1975359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:35.720130 2018 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: Bootstrap starting.
I20260619 00:59:35.721846 2018 tablet_bootstrap.cc:654] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.723522 2017 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: Bootstrap starting.
I20260619 00:59:35.724982 2018 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: No bootstrap required, opened a new log
I20260619 00:59:35.725144 2018 ts_tablet_manager.cc:1397] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 00:59:35.725159 2016 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: Bootstrap starting.
I20260619 00:59:35.725240 2017 tablet_bootstrap.cc:654] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.725956 2018 raft_consensus.cc:348] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.726619 2018 raft_consensus.cc:374] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.726692 2018 raft_consensus.cc:729] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfbe7832fdd43fa91336b7c78b09396, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.727064 2018 consensus_queue.cc:260] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.727499 2016 tablet_bootstrap.cc:654] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:35.727661 2023 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.728108 2023 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.731922 2018 ts_tablet_manager.cc:1428] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260619 00:59:35.734779 2023 raft_consensus.cc:482] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:35.734980 2023 raft_consensus.cc:504] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.736011 2017 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: No bootstrap required, opened a new log
I20260619 00:59:35.736222 2017 ts_tablet_manager.cc:1397] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260619 00:59:35.736804 2023 leader_election.cc:288] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367), d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257)
I20260619 00:59:35.737564 2017 raft_consensus.cc:348] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.737871 2017 raft_consensus.cc:374] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.738006 2017 raft_consensus.cc:729] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4dc474a5e8c4319a1117c1b4c574294, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.739100 2017 consensus_queue.cc:260] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.739604 2016 tablet_bootstrap.cc:492] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: No bootstrap required, opened a new log
I20260619 00:59:35.739859 2016 ts_tablet_manager.cc:1397] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260619 00:59:35.739655 2022 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.740024 2022 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.740780 2016 raft_consensus.cc:348] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.741397 2016 raft_consensus.cc:374] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:35.741516 2016 raft_consensus.cc:729] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d08fb92c32a94271b4682ab9e34c8bf9, State: Initialized, Role: FOLLOWER
I20260619 00:59:35.741641 2017 ts_tablet_manager.cc:1428] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260619 00:59:35.741816 2016 consensus_queue.cc:260] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.742333 2026 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.742688 2026 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.747233 2016 ts_tablet_manager.cc:1428] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260619 00:59:35.752012 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.737969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35257 (local address 127.0.97.1:41387)
0619 00:59:35.742394 (+ 4425us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.742417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.742677 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.743341 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.743347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.743374 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.743701 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.743716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.745324 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.745332 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.748784 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.748799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.751160 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.751167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.751338 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.751361 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.751491 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.751636 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4307,"thread_start_us":12674,"threads_started":1}
I20260619 00:59:35.752025 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.738438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41387 (local address 127.0.97.3:35257)
0619 00:59:35.738639 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.738646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.742499 (+ 3853us) server_negotiation.cc:407] Connection header received
0619 00:59:35.742965 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.742972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.743064 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.743132 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.743876 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.745056 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.749023 (+ 3967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.750895 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.751082 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.751209 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.751524 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.751682 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 00:59:35.752837 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed27ff0232364ada8bedd6d7b1975359" candidate_uuid: "fdfbe7832fdd43fa91336b7c78b09396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" is_pre_election: true
I20260619 00:59:35.753106 1952 raft_consensus.cc:2454] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfbe7832fdd43fa91336b7c78b09396 in term 0.
I20260619 00:59:35.753628 1599 leader_election.cc:302] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d08fb92c32a94271b4682ab9e34c8bf9, fdfbe7832fdd43fa91336b7c78b09396; no voters:
I20260619 00:59:35.754066 2023 raft_consensus.cc:2781] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:35.754196 2023 raft_consensus.cc:482] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:35.754317 2023 raft_consensus.cc:3037] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.757534 2023 raft_consensus.cc:504] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.758219 2023 leader_election.cc:288] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [CANDIDATE]: Term 1 election: Requested vote from peers f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367), d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257)
I20260619 00:59:35.758635 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed27ff0232364ada8bedd6d7b1975359" candidate_uuid: "fdfbe7832fdd43fa91336b7c78b09396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d08fb92c32a94271b4682ab9e34c8bf9"
I20260619 00:59:35.758903 1952 raft_consensus.cc:3037] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.760059 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.738498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45367 (local address 127.0.97.1:40401)
0619 00:59:35.751069 (+ 12571us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.751097 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.751323 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.751809 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.751814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.751840 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:35.752505 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.752516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.754116 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.754122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.756422 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.757210 (+ 788us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.759131 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.759138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.759369 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.759404 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.759572 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.759722 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12414,"thread_start_us":6137,"threads_started":1}
I20260619 00:59:35.760085 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.738839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40401 (local address 127.0.97.2:45367)
0619 00:59:35.739023 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.739030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.751194 (+ 12164us) server_negotiation.cc:407] Connection header received
0619 00:59:35.751492 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.751499 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.751572 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.751648 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:35.752888 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.753983 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.757410 (+ 3427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.758753 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.758954 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.759083 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.759619 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.759764 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 00:59:35.761101 1813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed27ff0232364ada8bedd6d7b1975359" candidate_uuid: "fdfbe7832fdd43fa91336b7c78b09396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc474a5e8c4319a1117c1b4c574294" is_pre_election: true
I20260619 00:59:35.761384 1813 raft_consensus.cc:2454] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfbe7832fdd43fa91336b7c78b09396 in term 0.
I20260619 00:59:35.762126 1952 raft_consensus.cc:2454] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfbe7832fdd43fa91336b7c78b09396 in term 1.
I20260619 00:59:35.762173 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed27ff0232364ada8bedd6d7b1975359" candidate_uuid: "fdfbe7832fdd43fa91336b7c78b09396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4dc474a5e8c4319a1117c1b4c574294"
I20260619 00:59:35.762458 1812 raft_consensus.cc:3037] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:35.762601 1599 leader_election.cc:302] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d08fb92c32a94271b4682ab9e34c8bf9, fdfbe7832fdd43fa91336b7c78b09396; no voters:
I20260619 00:59:35.762889 2023 raft_consensus.cc:2781] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:35.763440 2023 raft_consensus.cc:686] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 LEADER]: Becoming Leader. State: Replica: fdfbe7832fdd43fa91336b7c78b09396, State: Running, Role: LEADER
I20260619 00:59:35.764046 2023 consensus_queue.cc:237] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } }
I20260619 00:59:35.766434 1812 raft_consensus.cc:2454] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfbe7832fdd43fa91336b7c78b09396 in term 1.
I20260619 00:59:35.768494 2060 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: TxnStatusTablet state changed. Reason: New leader fdfbe7832fdd43fa91336b7c78b09396. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.769055 2060 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:35.770651 1512 catalog_manager.cc:5495] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } health_report { overall_health: UNKNOWN } } }
I20260619 00:59:35.776144 2062 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 00:59:35.776412 2062 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 00:59:35.795063 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.786981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.97.1:40569)
0619 00:59:35.787147 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.787153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.787239 (+ 86us) server_negotiation.cc:407] Connection header received
0619 00:59:35.787548 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.787554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.787669 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.787860 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:35.788262 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.789484 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.791533 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.792715 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.792850 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.793482 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.794301 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.794451 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.794629 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.794752 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 00:59:35.795200 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.786741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.0.1:40286)
0619 00:59:35.787121 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.787142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.787308 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.787887 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.787892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.787914 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:35.788106 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.788116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.789657 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.789663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.791348 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.791365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.792917 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.792925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.793168 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.793260 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.794442 (+ 1182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.794472 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.794586 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.794830 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260619 00:59:35.800221 1812 raft_consensus.cc:1261] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Refusing update from remote peer fdfbe7832fdd43fa91336b7c78b09396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.800519 1952 raft_consensus.cc:1261] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Refusing update from remote peer fdfbe7832fdd43fa91336b7c78b09396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.801290 2023 consensus_queue.cc:1035] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:35.801968 2023 consensus_queue.cc:1035] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:35.816433 2026 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: TxnStatusTablet state changed. Reason: New leader fdfbe7832fdd43fa91336b7c78b09396. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.816787 2026 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.817235 2022 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: TxnStatusTablet state changed. Reason: New leader fdfbe7832fdd43fa91336b7c78b09396. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.817555 2022 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.818305 2060 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.818532 2060 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:35.820102 2022 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.820381 2022 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P f4dc474a5e8c4319a1117c1b4c574294: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.823714 2023 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.824067 2023 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P fdfbe7832fdd43fa91336b7c78b09396: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:35.825368 2026 tablet_replica.cc:439] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fdfbe7832fdd43fa91336b7c78b09396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 } } peers { permanent_uuid: "f4dc474a5e8c4319a1117c1b4c574294" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45367 } } peers { permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 } } }
I20260619 00:59:35.825675 2026 tablet_replica.cc:442] T ed27ff0232364ada8bedd6d7b1975359 P d08fb92c32a94271b4682ab9e34c8bf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:35.838807 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.830258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.0.97.2:45367)
0619 00:59:35.830446 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.830453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.830474 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:59:35.830588 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.830596 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.830741 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.830935 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:35.831724 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.833001 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.835403 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.836079 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.836245 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.837192 (+ 947us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.837982 (+ 790us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.838143 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.838298 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.838395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 00:59:35.862318 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.852122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59359 (local address 127.0.97.1:40569)
0619 00:59:35.852374 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.852381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.855436 (+ 3055us) server_negotiation.cc:407] Connection header received
0619 00:59:35.855664 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.855671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.855758 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.855865 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:35.856512 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.857641 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.859917 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.860737 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.860878 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.861452 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.861607 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.861704 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.861893 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.861984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:59:35.862578 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.852111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40569 (local address 127.0.97.2:59359)
0619 00:59:35.855291 (+ 3180us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.855314 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.855502 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.855901 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.855906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.855921 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:35.856288 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.856301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.857902 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.857908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.859744 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.859761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.860945 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.860953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.861162 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.861244 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.861730 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.861743 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.861860 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.862024 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2967,"thread_start_us":2788,"threads_started":1}
I20260619 00:59:35.874356 1951 raft_consensus.cc:1261] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Refusing update from remote peer f4dc474a5e8c4319a1117c1b4c574294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.875227 2046 consensus_queue.cc:1035] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d08fb92c32a94271b4682ab9e34c8bf9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:35.875799 1674 raft_consensus.cc:1261] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Refusing update from remote peer f4dc474a5e8c4319a1117c1b4c574294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:35.876416 2046 consensus_queue.cc:1035] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfbe7832fdd43fa91336b7c78b09396" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:35.932261 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.922025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36645 (local address 127.0.97.2:45367)
0619 00:59:35.922200 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:35.922207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:35.923555 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 00:59:35.923832 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:35.923837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:35.923931 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:35.923991 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:35.925445 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.926571 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.929604 (+ 3033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.930326 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.930493 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.931426 (+ 933us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.931575 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.931639 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:35.931851 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 00:59:35.931948 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:59:35.933491 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:35.921600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45367 (local address 127.0.97.1:36645)
0619 00:59:35.923456 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:35.923476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:35.923678 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:35.924181 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:35.924185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:35.924199 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:35.925124 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.925135 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.927175 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.927180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.929409 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:35.929426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:35.930614 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:35.930621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:35.931108 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:35.931198 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:35.931717 (+ 519us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:35.931727 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 00:59:35.932928 (+ 1201us) client_negotiation.cc:240] Negotiation successful
0619 00:59:35.933088 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1652,"thread_start_us":1532,"threads_started":1}
I20260619 00:59:35.983011 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:35.983286 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:35.986765 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f36a4d0899764ae7bae15d5edfc26f07: Sending DeleteTablet for 3 replicas of tablet 39c0e8207ef94cba89ac4aa83bd0b179
I20260619 00:59:35.987813 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33128
I20260619 00:59:35.987941 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40284
I20260619 00:59:35.988631 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 39c0e8207ef94cba89ac4aa83bd0b179 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37998
I20260619 00:59:35.996534 2078 tablet_replica.cc:330] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: stopping tablet replica
I20260619 00:59:35.996909 2078 raft_consensus.cc:2229] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:35.997437 2078 raft_consensus.cc:2258] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:35.998972 2078 ts_tablet_manager.cc:1905] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:36.002897 2079 tablet_replica.cc:330] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: stopping tablet replica
I20260619 00:59:36.003662 2079 raft_consensus.cc:2229] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:36.004491 2079 raft_consensus.cc:2258] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:36.005957 2079 ts_tablet_manager.cc:1905] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:36.007716 2077 tablet_replica.cc:330] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: stopping tablet replica
I20260619 00:59:36.008034 2077 raft_consensus.cc:2229] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:36.008294 2077 raft_consensus.cc:2258] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:36.009505 2077 ts_tablet_manager.cc:1905] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:36.017081 2077 ts_tablet_manager.cc:1918] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:36.017289 2077 log.cc:1196] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/39c0e8207ef94cba89ac4aa83bd0b179
I20260619 00:59:36.017835 2077 ts_tablet_manager.cc:1939] T 39c0e8207ef94cba89ac4aa83bd0b179 P d08fb92c32a94271b4682ab9e34c8bf9: Deleting consensus metadata
I20260619 00:59:36.019114 1496 catalog_manager.cc:4841] TS d08fb92c32a94271b4682ab9e34c8bf9 (127.0.97.3:35257): tablet 39c0e8207ef94cba89ac4aa83bd0b179 (table test-table [id=f40b16979fa54d54b101d0718055d4ed]) successfully deleted
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:36.023837 2079 ts_tablet_manager.cc:1918] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:36.024078 2079 log.cc:1196] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/39c0e8207ef94cba89ac4aa83bd0b179
I20260619 00:59:36.024580 2079 ts_tablet_manager.cc:1939] T 39c0e8207ef94cba89ac4aa83bd0b179 P f4dc474a5e8c4319a1117c1b4c574294: Deleting consensus metadata
I20260619 00:59:36.024725 2078 ts_tablet_manager.cc:1918] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:36.024919 2078 log.cc:1196] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/39c0e8207ef94cba89ac4aa83bd0b179
I20260619 00:59:36.025841 1499 catalog_manager.cc:4841] TS f4dc474a5e8c4319a1117c1b4c574294 (127.0.97.2:45367): tablet 39c0e8207ef94cba89ac4aa83bd0b179 (table test-table [id=f40b16979fa54d54b101d0718055d4ed]) successfully deleted
I20260619 00:59:36.026052 2078 ts_tablet_manager.cc:1939] T 39c0e8207ef94cba89ac4aa83bd0b179 P fdfbe7832fdd43fa91336b7c78b09396: Deleting consensus metadata
I20260619 00:59:36.027161 1497 catalog_manager.cc:4841] TS fdfbe7832fdd43fa91336b7c78b09396 (127.0.97.1:40569): tablet 39c0e8207ef94cba89ac4aa83bd0b179 (table test-table [id=f40b16979fa54d54b101d0718055d4ed]) successfully deleted
I20260619 00:59:36.029301 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:36.002254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.0.97.1:40569)
0619 00:59:36.002493 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:36.002502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:36.009143 (+ 6641us) server_negotiation.cc:407] Connection header received
0619 00:59:36.009448 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:36.009454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:36.009552 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:36.009663 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:36.011791 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.012951 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.014844 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.015537 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.015637 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:36.020954 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:36.020969 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:36.020991 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:36.021040 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:36.025447 (+ 4407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.026798 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.026808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.026816 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.027154 (+ 338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.027660 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.027665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.027669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.028056 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:36.028371 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:36.028690 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 00:59:36.028915 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 00:59:36.030016 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40294
I20260619 00:59:36.057646 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:36.043984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.0.97.62:38955)
0619 00:59:36.044201 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:36.044208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:36.044671 (+ 463us) server_negotiation.cc:407] Connection header received
0619 00:59:36.044886 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:36.044893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:36.044989 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:36.045096 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:36.046464 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.047259 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.048496 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.049079 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.049219 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:36.052117 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:36.052124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:36.052128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:36.052171 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:36.055139 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.055772 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.055778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.055782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.055912 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.056287 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.056292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.056295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.056574 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:36.056818 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:36.057131 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 00:59:36.057343 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 00:59:36.058439 1512 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44858
I20260619 00:59:36.067564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1583
I20260619 00:59:36.094750 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:36.069185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.97.1:40569)
0619 00:59:36.069373 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:36.069379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:36.078451 (+ 9072us) server_negotiation.cc:407] Connection header received
0619 00:59:36.078876 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:36.078883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:36.078962 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:36.079029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:36.080736 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.081931 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.083765 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.084514 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.084604 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:36.087586 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:36.087593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:36.087598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:36.087633 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:36.091622 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.092458 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.092467 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.092471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.092597 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.093037 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.093042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.093046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.093368 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:36.093600 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:36.094106 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 00:59:36.094331 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 00:59:36.096284 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40308
W20260619 00:59:36.096822 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:36.097007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1583
I20260619 00:59:36.121619 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1723
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:36.140548 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:36.122409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.0.97.2:45367)
0619 00:59:36.122600 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:36.122608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:36.122683 (+ 75us) server_negotiation.cc:407] Connection header received
0619 00:59:36.122887 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:36.122892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:36.122987 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:36.123096 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:36.124544 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.125682 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.127561 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.128225 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.128358 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:36.133341 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:36.133357 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:36.133376 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:36.133431 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:36.137419 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.138218 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.138227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.138231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.138390 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.138847 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.138853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.138856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.139295 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:36.139624 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:36.139926 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 00:59:36.140152 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 00:59:36.141252 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38002
W20260619 00:59:36.141952 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:36.142119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1723
I20260619 00:59:36.160553 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1862
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:36.176086 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:36.161300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.0.97.3:35257)
0619 00:59:36.161475 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:36.161482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:36.161588 (+ 106us) server_negotiation.cc:407] Connection header received
0619 00:59:36.161785 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:36.161796 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:36.161867 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:36.161969 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:36.163175 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.163854 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.165016 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:36.165425 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:36.165555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:36.169632 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:36.169645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:36.169661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:36.169699 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:36.173090 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.173977 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.173984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.174020 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.174189 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:36.174581 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:36.174584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:36.174586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:36.174985 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:36.175229 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:36.175602 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 00:59:36.175778 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 00:59:36.176785 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33138
W20260619 00:59:36.177446 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:36.177628 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1862
I20260619 00:59:36.199249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 1480
I20260619 00:59:36.200367 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:36.312378 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 1480
2026-06-19T00:59:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4581 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[2088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[2088](info): set up 2 sockets
Jun 19 00:59:36 dist-test-slave-3rxd krb5kdc[2088](info): commencing operation
krb5kdc: starting...
W20260619 00:59:37.390862 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:37 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:37.582460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:37.910905 2101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:37.911180 2101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:37.911319 2101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:37.918629 2101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:37.918781 2101 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 00:59:37.918880 2101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:37.918991 2101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:37.919067 2101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:37.919147 2101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:37.928830 2101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45499
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:37.930110 2101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:37.931481 2101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:37.943315 2107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:37.943351 2109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:37.943274 2106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:37.944115 2101 server_base.cc:1034] running on GCE node
I20260619 00:59:37.944860 2101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:37.946245 2101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:37.947489 2101 hybrid_clock.cc:648] HybridClock initialized: now 1781830777947426 us; error 39 us; skew 500 ppm
I20260619 00:59:37.947813 2101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:37 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:37.950881 2101 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:37.952310 2101 webserver.cc:460] Webserver started at http://127.0.97.62:45509/ using document root <none> and password file <none>
I20260619 00:59:37.952991 2101 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:37.953114 2101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:37.953424 2101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:37.956317 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "24d33cf6934b4c7598285befc368eafe"
format_stamp: "Formatted at 2026-06-19 00:59:37 on dist-test-slave-3rxd"
server_key: "5511929c5640fa0a4ce09fc0db4613d4"
server_key_iv: "2103fc0675957732f925d99eb8431a4d"
server_key_version: "encryptionkey@0"
I20260619 00:59:37.957064 2101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "24d33cf6934b4c7598285befc368eafe"
format_stamp: "Formatted at 2026-06-19 00:59:37 on dist-test-slave-3rxd"
server_key: "5511929c5640fa0a4ce09fc0db4613d4"
server_key_iv: "2103fc0675957732f925d99eb8431a4d"
server_key_version: "encryptionkey@0"
I20260619 00:59:37.961951 2101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 00:59:37.965260 2116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:37.966572 2101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:59:37.966799 2101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "24d33cf6934b4c7598285befc368eafe"
format_stamp: "Formatted at 2026-06-19 00:59:37 on dist-test-slave-3rxd"
server_key: "5511929c5640fa0a4ce09fc0db4613d4"
server_key_iv: "2103fc0675957732f925d99eb8431a4d"
server_key_version: "encryptionkey@0"
I20260619 00:59:37.967023 2101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:37.984107 2101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:37.987761 2101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:37.988037 2101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:38.008490 2101 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45499
I20260619 00:59:38.008522 2177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45499 every 8 connection(s)
I20260619 00:59:38.009850 2101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:38.013572 2178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:38.020010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2101
I20260619 00:59:38.020229 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:38.021034 388 external_mini_cluster.cc:1428] Setting key 7f3bb8b67c6ad02066cab5eaf16c39fe
I20260619 00:59:38.024585 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Bootstrap starting.
I20260619 00:59:38.028009 2178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:38.029045 2178 log.cc:824] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:38.031699 2178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: No bootstrap required, opened a new log
Jun 19 00:59:38 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:38.039443 2178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260619 00:59:38.039829 2178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:38.039947 2178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d33cf6934b4c7598285befc368eafe, State: Initialized, Role: FOLLOWER
I20260619 00:59:38.040578 2178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260619 00:59:38.040810 2178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:38.040982 2178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:38.041177 2178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:38.042428 2181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:38.023082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.0.97.62:45499)
0619 00:59:38.023864 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:38.023876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:38.023915 (+ 39us) server_negotiation.cc:407] Connection header received
0619 00:59:38.025664 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:38.025684 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:38.026081 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:38.026405 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:38.028152 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.029482 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.030676 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.031376 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.031556 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:38.035329 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:38.035336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:38.035354 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:38.035390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:38.038441 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.039153 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.039160 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.039166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.039312 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.039680 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.039689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.039691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.040329 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:38.040622 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:38.041039 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 00:59:38.041436 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":388,"threads_started":1}
I20260619 00:59:38.043615 2178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260619 00:59:38.044096 2178 leader_election.cc:302] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24d33cf6934b4c7598285befc368eafe; no voters:
I20260619 00:59:38.044694 2178 leader_election.cc:288] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:38.044840 2183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:38.045841 2183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [term 1 LEADER]: Becoming Leader. State: Replica: 24d33cf6934b4c7598285befc368eafe, State: Running, Role: LEADER
I20260619 00:59:38.046320 2183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } }
I20260619 00:59:38.046826 2178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:38.049072 2185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24d33cf6934b4c7598285befc368eafe. Latest consensus state: current_term: 1 leader_uuid: "24d33cf6934b4c7598285befc368eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260619 00:59:38.049371 2185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:38.049801 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24d33cf6934b4c7598285befc368eafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24d33cf6934b4c7598285befc368eafe" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45499 } } }
I20260619 00:59:38.050101 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:38.051224 2190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:38.056277 2190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:38.065287 2190 catalog_manager.cc:1347] Generated new cluster ID: ea2b0961cc834e7aa4d65c2619be11bc
I20260619 00:59:38.065404 2190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:38.082810 2190 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:38.084040 2190 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:38.098590 2190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Generated new TSK 0
I20260619 00:59:38.099429 2190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:38.153784 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:38.480240 2204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:38.480523 2204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:38.480727 2204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:38.487766 2204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:38.487896 2204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:38.488061 2204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:59:38.488138 2204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:38.488265 2204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:38.498682 2204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:38.500121 2204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:38.501427 2204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:38.514812 2209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:38.515018 2212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:38.515044 2210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:38.515646 2204 server_base.cc:1034] running on GCE node
I20260619 00:59:38.516291 2204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:38.517189 2204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:38.518420 2204 hybrid_clock.cc:648] HybridClock initialized: now 1781830778518354 us; error 55 us; skew 500 ppm
I20260619 00:59:38.518702 2204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:38 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:38.521801 2204 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:38.523350 2204 webserver.cc:460] Webserver started at http://127.0.97.1:34255/ using document root <none> and password file <none>
I20260619 00:59:38.524039 2204 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:38.524196 2204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:38.524521 2204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:38.527535 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "32784c536d4e446ea59d2eaaff19b24f"
format_stamp: "Formatted at 2026-06-19 00:59:38 on dist-test-slave-3rxd"
server_key: "10362f1f798162ae4c14b8f43bc955f4"
server_key_iv: "85fc04adc7125e94e94bdfe2f32cec06"
server_key_version: "encryptionkey@0"
I20260619 00:59:38.528317 2204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "32784c536d4e446ea59d2eaaff19b24f"
format_stamp: "Formatted at 2026-06-19 00:59:38 on dist-test-slave-3rxd"
server_key: "10362f1f798162ae4c14b8f43bc955f4"
server_key_iv: "85fc04adc7125e94e94bdfe2f32cec06"
server_key_version: "encryptionkey@0"
I20260619 00:59:38.533463 2204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:59:38.536919 2219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:38.538267 2204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 00:59:38.538477 2204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "32784c536d4e446ea59d2eaaff19b24f"
format_stamp: "Formatted at 2026-06-19 00:59:38 on dist-test-slave-3rxd"
server_key: "10362f1f798162ae4c14b8f43bc955f4"
server_key_iv: "85fc04adc7125e94e94bdfe2f32cec06"
server_key_version: "encryptionkey@0"
I20260619 00:59:38.538683 2204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:38.566181 2204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:38.570030 2204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:38.570323 2204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:38.573071 2204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:38.573225 2204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:38.573379 2204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:38.573493 2204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:59:38 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:38.594753 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:38.574927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55459 (local address 127.0.97.62:45499)
0619 00:59:38.576160 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:38.576167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:38.576190 (+ 23us) server_negotiation.cc:407] Connection header received
0619 00:59:38.577633 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:38.577638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:38.577727 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:38.577851 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:38.580072 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.581413 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.583167 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.583558 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.583689 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:38.587814 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:38.587818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:38.587821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:38.587853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:38.590961 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.592224 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.592229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.592233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.592404 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.592955 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.592960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.592965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.593240 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:38.593405 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:38.594118 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 00:59:38.594346 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":995,"threads_started":1}
I20260619 00:59:38.595798 2228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:38.574437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:55459)
0619 00:59:38.575412 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:38.575468 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:38.577365 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:38.578171 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:38.578188 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:38.579010 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:38.579840 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:38.579878 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.581593 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.581598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:38.583002 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:38.583029 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.583779 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.583788 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:38.584008 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:38.584773 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:38.584812 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:38.587591 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:38.591150 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:38.591159 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:38.591181 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:38.592035 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:38.592546 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:38.592551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:38.592556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:38.592813 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:38.593377 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:38.593385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:38.593762 (+ 377us) client_negotiation.cc:769] Sending connection context
0619 00:59:38.594031 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 00:59:38.594613 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":484,"threads_started":1}
I20260619 00:59:38.616899 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:38.601967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38249 (local address 127.0.97.62:45499)
0619 00:59:38.602157 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:38.602164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:38.602960 (+ 796us) server_negotiation.cc:407] Connection header received
0619 00:59:38.603086 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:38.603094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:38.603201 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:38.603324 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:38.604217 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.605675 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.607054 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.607601 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.607814 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:38.610605 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:38.610610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:38.610613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:38.610644 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:38.613903 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.614897 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.614901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.614903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.615026 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:38.615381 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:38.615387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:38.615392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:38.615779 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:38.615921 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:38.616337 (+ 416us) server_negotiation.cc:299] Negotiation successful
0619 00:59:38.616559 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:59:38.616997 2281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:38.602075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.1:38249)
0619 00:59:38.602848 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:38.602868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:38.602983 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:38.603353 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:38.603356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:38.603793 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:38.604077 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:38.604086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.605813 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.605817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:38.606876 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:38.606886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:38.607886 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:38.607895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:38.608217 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:38.609030 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:38.609050 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:38.610365 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:38.614092 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:38.614097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:38.614101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:38.614743 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:38.615138 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:38.615142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:38.615150 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:38.615266 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:38.615885 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:38.615889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:38.616082 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 00:59:38.616233 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 00:59:38.616544 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":441,"threads_started":1}
I20260619 00:59:38.628795 2204 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33077
I20260619 00:59:38.628829 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33077 every 8 connection(s)
I20260619 00:59:38.630038 2204 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:38.631991 2340 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260619 00:59:38.632217 2340 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:38.632839 2340 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260619 00:59:38.634678 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2204
I20260619 00:59:38.634824 2133 ts_manager.cc:194] Registered new tserver with Master: 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077)
I20260619 00:59:38.634922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:38.635506 388 external_mini_cluster.cc:1428] Setting key 3a1c053553ab4884663e92de11e37fde
I20260619 00:59:38.636503 2133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55459
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:38.676131 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:38.994204 2344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:38.994493 2344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:38.994674 2344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:39.002103 2344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:39.002231 2344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:39.002395 2344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:59:39.002478 2344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:39.002588 2344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:39.012212 2344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:39.013392 2344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:39.014617 2344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:39.027290 2352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:39.027436 2349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:39.027447 2350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:39.028301 2344 server_base.cc:1034] running on GCE node
I20260619 00:59:39.028872 2344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:39.029774 2344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:39.031003 2344 hybrid_clock.cc:648] HybridClock initialized: now 1781830779030954 us; error 64 us; skew 500 ppm
I20260619 00:59:39.031307 2344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:39.037802 2344 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:39.039269 2344 webserver.cc:460] Webserver started at http://127.0.97.2:38205/ using document root <none> and password file <none>
I20260619 00:59:39.039983 2344 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:39.040124 2344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:39.040421 2344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:39.043361 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "bb7c979727bd4df29403bc42ae802e83"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "a9db7f6ac043390693753d0d18dedf53"
server_key_iv: "981680bf29fe7154b3b150ed286e3a75"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.044106 2344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bb7c979727bd4df29403bc42ae802e83"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "a9db7f6ac043390693753d0d18dedf53"
server_key_iv: "981680bf29fe7154b3b150ed286e3a75"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.049149 2344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:39.052415 2359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:39.053654 2344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:59:39.053821 2344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "bb7c979727bd4df29403bc42ae802e83"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "a9db7f6ac043390693753d0d18dedf53"
server_key_iv: "981680bf29fe7154b3b150ed286e3a75"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.054023 2344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:39.068476 2344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:39.072647 2344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:39.072923 2344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:39.075618 2344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:39.075739 2344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:39.075881 2344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:39.076021 2344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:39.097651 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.076052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35475 (local address 127.0.97.62:45499)
0619 00:59:39.076230 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.076234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.077718 (+ 1484us) server_negotiation.cc:407] Connection header received
0619 00:59:39.079773 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.079779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.079874 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.079994 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:39.082201 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.083356 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.084976 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.085366 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.085505 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.090203 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:39.090213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:39.090220 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:39.090275 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:39.093797 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.094877 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.094883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.094887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.095084 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.095747 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.095753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.095756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.096078 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:39.096240 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.097084 (+ 844us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.097306 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 00:59:39.098875 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.076718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:35475)
0619 00:59:39.077554 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.077598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.079487 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.080236 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.080245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.081034 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:39.081991 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.082018 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.083502 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.083508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.084804 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.084816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.085598 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.085607 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.085826 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.086521 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:39.086542 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:39.089876 (+ 3334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:39.094009 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.094020 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.094044 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.094705 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.095222 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.095229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.095235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.095584 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.096217 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:39.096229 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:39.096661 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.096993 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.097558 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260619 00:59:39.122885 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.104988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44259 (local address 127.0.97.62:45499)
0619 00:59:39.105147 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.105154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.106275 (+ 1121us) server_negotiation.cc:407] Connection header received
0619 00:59:39.106482 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.106489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.106577 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.106678 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:39.108344 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.109578 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.112154 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.112893 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.113039 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.115767 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:39.115771 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:39.115774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:39.115802 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:39.119116 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.120307 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.120313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.120316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.120473 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.120965 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.120971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.120977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.121463 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:39.121627 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.122233 (+ 606us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.122512 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 00:59:39.123059 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.105293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.2:44259)
0619 00:59:39.106165 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.106183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.106336 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.106769 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.106778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.107598 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:39.108150 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.108164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.109753 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.109762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.111988 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.112011 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.113137 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.113147 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.113376 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.114131 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:39.114174 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:39.115604 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:39.119276 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.119283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.119289 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.120158 (+ 869us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.120604 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.120611 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.120618 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.120815 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.121604 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:39.121612 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:39.121918 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.122092 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.122563 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":536,"threads_started":1}
I20260619 00:59:39.125219 2344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44149
I20260619 00:59:39.125299 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44149 every 8 connection(s)
I20260619 00:59:39.126539 2344 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:39.128846 2479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260619 00:59:39.129062 2479 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:39.129684 2479 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260619 00:59:39.130914 2132 ts_manager.cc:194] Registered new tserver with Master: bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149)
I20260619 00:59:39.131848 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35475
I20260619 00:59:39.134084 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2344
I20260619 00:59:39.134275 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:39.134785 388 external_mini_cluster.cc:1428] Setting key 83f15540ea69132cb95f172732f4f579
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:39.173492 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45499
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:39.488334 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:39.488598 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:39.488756 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:39.495898 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:39.495998 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:39.496181 2483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 00:59:39.496256 2483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 00:59:39.496361 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:39.505995 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:45499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:39.507308 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:39.508910 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:39.525566 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:39.525491 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:39.525545 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:39.526356 2483 server_base.cc:1034] running on GCE node
I20260619 00:59:39.527099 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:39.528134 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:39.529373 2483 hybrid_clock.cc:648] HybridClock initialized: now 1781830779529321 us; error 67 us; skew 500 ppm
I20260619 00:59:39.529733 2483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:39.532814 2483 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:39.534291 2483 webserver.cc:460] Webserver started at http://127.0.97.3:34767/ using document root <none> and password file <none>
I20260619 00:59:39.535040 2483 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:39.535181 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:39.535491 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:39.538329 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "870afa83a8b87e83cc7749f4f095da3a"
server_key_iv: "8985af1de374982703e5c31703495bf9"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.539331 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "870afa83a8b87e83cc7749f4f095da3a"
server_key_iv: "8985af1de374982703e5c31703495bf9"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.544500 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:39.548060 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:39.549302 2483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:59:39.549480 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
format_stamp: "Formatted at 2026-06-19 00:59:39 on dist-test-slave-3rxd"
server_key: "870afa83a8b87e83cc7749f4f095da3a"
server_key_iv: "8985af1de374982703e5c31703495bf9"
server_key_version: "encryptionkey@0"
I20260619 00:59:39.549687 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:39.596560 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:39.600896 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:39.601210 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:39.604094 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:39.604224 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:39.604386 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:39.604522 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:39.626958 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.604507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34139 (local address 127.0.97.62:45499)
0619 00:59:39.604668 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.604672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.606087 (+ 1415us) server_negotiation.cc:407] Connection header received
0619 00:59:39.608126 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.608131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.608225 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.608333 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:39.610493 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.611901 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.613518 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.614379 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.614553 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.619263 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:39.619276 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:39.619279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:39.619332 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:39.622818 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.623844 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.623850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.623853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.624022 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.624568 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.624574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.624578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.624985 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:39.625137 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.626083 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.626380 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 00:59:39.627633 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.605163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:34139)
0619 00:59:39.605914 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.605961 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.607853 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.608567 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.608575 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.609251 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:39.610292 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.610324 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.612114 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.612118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.613368 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.613378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.614597 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.614601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.614830 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.615569 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:39.615602 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:39.619043 (+ 3441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:39.622962 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.622970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.622989 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.623715 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.624153 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.624160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.624165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.624448 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.625099 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:39.625106 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:39.625603 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.625943 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.626399 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":329,"threads_started":1}
I20260619 00:59:39.639369 2340 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260619 00:59:39.654091 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.639925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51111 (local address 127.0.97.62:45499)
0619 00:59:39.640151 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.640159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.640865 (+ 706us) server_negotiation.cc:407] Connection header received
0619 00:59:39.641051 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.641057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.641133 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.641227 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:39.642195 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.643070 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.645000 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.645440 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.645586 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.648226 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:39.648235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:39.648240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:39.648279 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:39.650911 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.651881 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.651890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.651896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.652054 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.652547 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.652555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.652558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.652864 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:39.653003 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.653566 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.653788 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 00:59:39.654582 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.639816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.97.3:51111)
0619 00:59:39.640729 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.640754 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.640923 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.641282 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.641287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.641793 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:39.642059 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.642068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.643252 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.643261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.644867 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.644879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.645667 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.645677 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.645928 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.646732 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:39.646763 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:39.648016 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:39.651036 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.651042 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.651046 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.651620 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.652180 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:39.652188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:39.652193 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:39.652352 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:39.653013 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:39.653019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:39.653235 (+ 216us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.653438 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.653935 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":535,"threads_started":1}
I20260619 00:59:39.665632 2483 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38607
I20260619 00:59:39.665663 2617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38607 every 8 connection(s)
I20260619 00:59:39.666989 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:39.668983 2618 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45499
I20260619 00:59:39.669198 2618 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:39.669801 2618 heartbeater.cc:511] Master 127.0.97.62:45499 requested a full tablet report, sending...
I20260619 00:59:39.670996 2132 ts_manager.cc:194] Registered new tserver with Master: f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:39.671936 2132 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34139
I20260619 00:59:39.672603 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2483
I20260619 00:59:39.672771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:39.673297 388 external_mini_cluster.cc:1428] Setting key ad20d0a9829254a9e65d63dedabff010
I20260619 00:59:39.676149 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:59:39 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:39.722163 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.708420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.0.97.62:45499)
0619 00:59:39.708625 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.708631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.708761 (+ 130us) server_negotiation.cc:407] Connection header received
0619 00:59:39.708992 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.708996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.709063 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.709173 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:39.710509 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.711293 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.712568 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.712977 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.713110 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.716774 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:39.716779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:39.716782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:39.716814 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:39.719435 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.720118 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.720124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.720129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.720275 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:39.720773 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:39.720779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:39.720782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:39.721036 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:39.721283 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.721591 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.721806 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 00:59:39.726075 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:39.733074 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:39.770018 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.746010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36172 (local address 127.0.97.3:38607)
0619 00:59:39.759113 (+ 13103us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.759125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.759557 (+ 432us) server_negotiation.cc:407] Connection header received
0619 00:59:39.759983 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.759995 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.760340 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.760573 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:39.761820 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.764222 (+ 2402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.767404 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.768800 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.768951 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.769198 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.769480 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.769656 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12920,"thread_start_us":12568,"threads_started":1}
I20260619 00:59:39.770083 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.745754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38607 (local address 127.0.0.1:36172)
0619 00:59:39.759397 (+ 13643us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.759430 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.759651 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.761015 (+ 1364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.761026 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.761069 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:39.761541 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.761559 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.764435 (+ 2876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.764443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.767063 (+ 2620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.767086 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.769019 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.769025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.769234 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.769281 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.769438 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.769637 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13405,"mutex_wait_us":51,"thread_start_us":14562,"threads_started":1}
I20260619 00:59:39.770848 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.745851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33077 (local address 127.0.0.1:41970)
0619 00:59:39.760128 (+ 14277us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.760151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.761384 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.762438 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.762446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.762479 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:39.762986 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.763000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.765778 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.765783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.767860 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.767871 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.769965 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.769972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.770198 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.770231 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.770301 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.770428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14121,"mutex_wait_us":48,"thread_start_us":13199,"threads_started":1}
I20260619 00:59:39.771080 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.746493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41970 (local address 127.0.97.1:33077)
0619 00:59:39.755861 (+ 9368us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.755874 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.760274 (+ 4400us) server_negotiation.cc:407] Connection header received
0619 00:59:39.761633 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.761647 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.761940 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.762128 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:39.763205 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.765640 (+ 2435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.768031 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.769497 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.769633 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.769820 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.770579 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.770774 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9139,"thread_start_us":8976,"threads_started":1}
I20260619 00:59:39.773943 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.746260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44149 (local address 127.0.0.1:46356)
0619 00:59:39.765297 (+ 19037us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:39.765319 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:39.765429 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:39.766267 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:39.766270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:39.766290 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:39.766693 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.766701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.768435 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.768440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.770568 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:39.770580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.773190 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.773197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:39.773380 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.773410 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:59:39.773502 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 00:59:39.773616 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18917,"mutex_wait_us":25,"thread_start_us":18771,"threads_started":1}
I20260619 00:59:39.773958 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:39.746990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.97.2:44149)
0619 00:59:39.764668 (+ 17678us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:39.764677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:39.765541 (+ 864us) server_negotiation.cc:407] Connection header received
0619 00:59:39.765648 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:39.765656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:39.765878 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:39.766111 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:39.766925 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.768290 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.770772 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:39.772997 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:39.773133 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:39.773345 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:39.773556 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 00:59:39.773701 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17509,"thread_start_us":17684,"threads_started":1}
I20260619 00:59:39.774401 2548 tablet_service.cc:1467] Processing CreateTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 (DEFAULT_TABLE table=test-table [id=75bba46bf63c47f6b58d6bf35ed73228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:39.774922 2272 tablet_service.cc:1467] Processing CreateTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 (DEFAULT_TABLE table=test-table [id=75bba46bf63c47f6b58d6bf35ed73228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:39.776120 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25c0dce71ec4998842ce597fd0a0a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:39.776988 2409 tablet_service.cc:1467] Processing CreateTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 (DEFAULT_TABLE table=test-table [id=75bba46bf63c47f6b58d6bf35ed73228]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:39.778061 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25c0dce71ec4998842ce597fd0a0a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:39.783296 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c25c0dce71ec4998842ce597fd0a0a90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:39.786687 2637 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Bootstrap starting.
I20260619 00:59:39.789754 2638 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Bootstrap starting.
I20260619 00:59:39.789846 2637 tablet_bootstrap.cc:654] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:39.791145 2637 log.cc:824] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:39.793022 2638 tablet_bootstrap.cc:654] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:39.794266 2637 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: No bootstrap required, opened a new log
I20260619 00:59:39.794353 2638 log.cc:824] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:39.794582 2637 ts_tablet_manager.cc:1397] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 00:59:39.797403 2638 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: No bootstrap required, opened a new log
I20260619 00:59:39.797730 2638 ts_tablet_manager.cc:1397] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 00:59:39.805352 2641 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Bootstrap starting.
I20260619 00:59:39.805379 2637 raft_consensus.cc:348] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.806051 2637 raft_consensus.cc:374] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:39.806169 2637 raft_consensus.cc:729] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Initialized, Role: FOLLOWER
I20260619 00:59:39.806828 2637 consensus_queue.cc:260] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.808717 2637 ts_tablet_manager.cc:1428] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 00:59:39.809257 2641 tablet_bootstrap.cc:654] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:39.809372 2638 raft_consensus.cc:348] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.809862 2638 raft_consensus.cc:374] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:39.810009 2638 raft_consensus.cc:729] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Initialized, Role: FOLLOWER
I20260619 00:59:39.810770 2638 consensus_queue.cc:260] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.811553 2641 log.cc:824] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:39.811992 2479 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260619 00:59:39.812692 2638 ts_tablet_manager.cc:1428] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260619 00:59:39.815754 2641 tablet_bootstrap.cc:492] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: No bootstrap required, opened a new log
I20260619 00:59:39.816064 2641 ts_tablet_manager.cc:1397] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 00:59:39.823526 2641 raft_consensus.cc:348] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.824088 2641 raft_consensus.cc:374] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:39.824210 2641 raft_consensus.cc:729] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4b1b6b5074610a2e3b95b953354b8, State: Initialized, Role: FOLLOWER
I20260619 00:59:39.824777 2641 consensus_queue.cc:260] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:39.825690 2618 heartbeater.cc:503] Master 127.0.97.62:45499 was elected leader, sending a full tablet report...
I20260619 00:59:39.826146 2641 ts_tablet_manager.cc:1428] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
W20260619 00:59:39.878744 2480 tablet.cc:2369] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:59:39.882441 2341 tablet.cc:2369] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 00:59:39.918663 2619 tablet.cc:2369] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 00:59:40.037624 2642 raft_consensus.cc:482] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:40.037922 2642 raft_consensus.cc:504] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:40.039868 2642 leader_election.cc:288] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607), bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149)
I20260619 00:59:40.048032 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.040373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54759 (local address 127.0.97.3:38607)
0619 00:59:40.040632 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.040642 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.041285 (+ 643us) server_negotiation.cc:407] Connection header received
0619 00:59:40.041510 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.041514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.041570 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.041665 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:40.042234 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.043058 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.045710 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.046986 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.047099 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.047209 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.047634 (+ 425us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.047757 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 00:59:40.048130 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.040308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37415 (local address 127.0.97.2:44149)
0619 00:59:40.040545 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.040554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.041410 (+ 856us) server_negotiation.cc:407] Connection header received
0619 00:59:40.041629 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.041634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.041694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.041796 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:40.042516 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.043367 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.045715 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.046958 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.047099 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.047206 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.047781 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.047908 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 00:59:40.048282 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.040204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44149 (local address 127.0.97.1:37415)
0619 00:59:40.041292 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:40.041315 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:40.041482 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:40.041840 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:40.041847 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:40.041901 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:40.042345 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.042358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.043541 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.043547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.045564 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.045576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.047414 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.047420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.047636 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.047665 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 00:59:40.047740 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 00:59:40.047836 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":20,"spinlock_wait_cycles":768,"thread_start_us":696,"threads_started":1}
I20260619 00:59:40.048403 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.040188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38607 (local address 127.0.97.1:54759)
0619 00:59:40.041178 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:40.041194 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:40.041361 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:40.041679 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:40.041683 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:40.041706 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:40.042088 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.042098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.043177 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.043185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.045504 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.045518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.047223 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.047229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.047420 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.047519 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 00:59:40.047612 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 00:59:40.047788 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":26,"spinlock_wait_cycles":21376,"thread_start_us":633,"threads_started":1}
I20260619 00:59:40.049425 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25c0dce71ec4998842ce597fd0a0a90" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" is_pre_election: true
I20260619 00:59:40.049856 2572 raft_consensus.cc:2454] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.050459 2220 leader_election.cc:302] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, f8f4b1b6b5074610a2e3b95b953354b8; no voters:
I20260619 00:59:40.050621 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25c0dce71ec4998842ce597fd0a0a90" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83" is_pre_election: true
I20260619 00:59:40.050884 2642 raft_consensus.cc:2781] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:40.051045 2642 raft_consensus.cc:482] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:40.051156 2642 raft_consensus.cc:3037] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.051169 2434 raft_consensus.cc:2454] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.053756 2642 raft_consensus.cc:504] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:40.054337 2642 leader_election.cc:288] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Requested vote from peers f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607), bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149)
I20260619 00:59:40.054800 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25c0dce71ec4998842ce597fd0a0a90" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83"
I20260619 00:59:40.054786 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c25c0dce71ec4998842ce597fd0a0a90" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
I20260619 00:59:40.054986 2572 raft_consensus.cc:3037] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.054987 2434 raft_consensus.cc:3037] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.058019 2572 raft_consensus.cc:2454] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.058038 2434 raft_consensus.cc:2454] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.058429 2220 leader_election.cc:302] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, f8f4b1b6b5074610a2e3b95b953354b8; no voters:
I20260619 00:59:40.058755 2642 raft_consensus.cc:2781] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:40.059171 2642 raft_consensus.cc:686] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Becoming Leader. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Running, Role: LEADER
I20260619 00:59:40.059612 2642 consensus_queue.cc:237] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } }
I20260619 00:59:40.063819 2132 catalog_manager.cc:5495] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } health_report { overall_health: UNKNOWN } } }
Jun 19 00:59:40 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:40.178257 2229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.165169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.0.97.62:45499)
0619 00:59:40.165324 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.165331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.165872 (+ 541us) server_negotiation.cc:407] Connection header received
0619 00:59:40.166078 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.166086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.166141 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.166254 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:40.167415 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.168153 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.169392 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.169814 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.169947 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.173266 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:40.173271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:40.173274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:40.173303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:40.175879 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:40.176533 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:40.176537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:40.176539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:40.176660 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:40.177038 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:40.177042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:40.177044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:40.177287 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:40.177397 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.177815 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.178012 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
I20260619 00:59:40.178265 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.165186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45499 (local address 127.0.0.1:55658)
0619 00:59:40.165770 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:40.165785 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:40.165954 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:40.166273 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:40.166277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:40.167043 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:40.167282 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.167291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.168273 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.168281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.169255 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.169266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.169999 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.170005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.170162 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.170845 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:40.170866 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:40.173092 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:40.175994 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:40.176003 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:40.176009 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:40.176421 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:40.176787 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:40.176791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:40.176793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:40.176933 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:40.177406 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:40.177412 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:40.177559 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 00:59:40.177695 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 00:59:40.177899 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":303,"threads_started":1}
I20260619 00:59:40.183389 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55658:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 00:59:40.184391 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:40.190405 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 161681821cf54205a547889a072db518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6dc4a2cef05a4e2e9eb03dbab7ad07ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:40.190465 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 161681821cf54205a547889a072db518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6dc4a2cef05a4e2e9eb03dbab7ad07ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:40.191438 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161681821cf54205a547889a072db518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.190986 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 161681821cf54205a547889a072db518 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6dc4a2cef05a4e2e9eb03dbab7ad07ee]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 00:59:40.191455 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161681821cf54205a547889a072db518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.191776 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 161681821cf54205a547889a072db518. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.196627 2641 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: Bootstrap starting.
I20260619 00:59:40.198082 2641 tablet_bootstrap.cc:654] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.198611 2638 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: Bootstrap starting.
I20260619 00:59:40.199091 2637 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: Bootstrap starting.
I20260619 00:59:40.200564 2637 tablet_bootstrap.cc:654] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.200639 2638 tablet_bootstrap.cc:654] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.202808 2637 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: No bootstrap required, opened a new log
I20260619 00:59:40.202842 2641 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: No bootstrap required, opened a new log
I20260619 00:59:40.202953 2637 ts_tablet_manager.cc:1397] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260619 00:59:40.202994 2641 ts_tablet_manager.cc:1397] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20260619 00:59:40.203588 2637 raft_consensus.cc:348] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.203612 2641 raft_consensus.cc:348] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.203761 2637 raft_consensus.cc:374] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.203812 2641 raft_consensus.cc:374] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.203855 2637 raft_consensus.cc:729] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.203908 2641 raft_consensus.cc:729] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4b1b6b5074610a2e3b95b953354b8, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.204079 2637 consensus_queue.cc:260] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.204124 2641 consensus_queue.cc:260] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.204465 2642 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.204677 2637 ts_tablet_manager.cc:1428] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 00:59:40.204701 2641 ts_tablet_manager.cc:1428] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 00:59:40.204771 2642 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.205157 2645 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.205435 2645 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.206014 2638 tablet_bootstrap.cc:492] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: No bootstrap required, opened a new log
I20260619 00:59:40.206193 2638 ts_tablet_manager.cc:1397] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260619 00:59:40.207294 2638 raft_consensus.cc:348] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.207536 2638 raft_consensus.cc:374] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.207655 2638 raft_consensus.cc:729] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.207893 2638 consensus_queue.cc:260] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.208454 2638 ts_tablet_manager.cc:1428] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 00:59:40.208323 2643 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.208729 2643 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.227783 2642 raft_consensus.cc:482] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:40.228014 2642 raft_consensus.cc:504] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.228732 2642 leader_election.cc:288] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:40.229130 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161681821cf54205a547889a072db518" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83" is_pre_election: true
I20260619 00:59:40.229331 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161681821cf54205a547889a072db518" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" is_pre_election: true
I20260619 00:59:40.229384 2434 raft_consensus.cc:2454] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.229588 2572 raft_consensus.cc:2454] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.229849 2221 leader_election.cc:302] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83; no voters:
I20260619 00:59:40.230137 2642 raft_consensus.cc:2781] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:40.230229 2642 raft_consensus.cc:482] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:40.230372 2642 raft_consensus.cc:3037] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.232399 2642 raft_consensus.cc:504] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.232967 2642 leader_election.cc:288] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Requested vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:40.233330 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161681821cf54205a547889a072db518" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83"
I20260619 00:59:40.233409 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "161681821cf54205a547889a072db518" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
I20260619 00:59:40.233557 2434 raft_consensus.cc:3037] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.233592 2572 raft_consensus.cc:3037] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.236475 2434 raft_consensus.cc:2454] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.236475 2572 raft_consensus.cc:2454] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.236811 2221 leader_election.cc:302] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83; no voters:
I20260619 00:59:40.237074 2642 raft_consensus.cc:2781] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:40.237237 2642 raft_consensus.cc:686] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Becoming Leader. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Running, Role: LEADER
I20260619 00:59:40.237480 2642 consensus_queue.cc:237] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.239130 2648 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: TxnStatusTablet state changed. Reason: New leader 32784c536d4e446ea59d2eaaff19b24f. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.239476 2648 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:40.240118 2660 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 00:59:40.240391 2660 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 00:59:40.240613 2133 catalog_manager.cc:5495] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } health_report { overall_health: UNKNOWN } } }
I20260619 00:59:40.262270 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.255529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33077 (local address 127.0.0.1:41984)
0619 00:59:40.255776 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:40.255795 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:40.255932 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:40.256447 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:40.256451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:40.256466 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:40.256637 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.256645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.258177 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.258181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.259772 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.259785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.260725 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.260729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.260890 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.260941 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:40.261873 (+ 932us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:40.261886 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 00:59:40.261936 (+ 50us) client_negotiation.cc:240] Negotiation successful
0619 00:59:40.262045 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 00:59:40.262356 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.255641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.0.97.1:33077)
0619 00:59:40.255803 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.255842 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.255860 (+ 18us) server_negotiation.cc:407] Connection header received
0619 00:59:40.256137 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.256142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.256242 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.256412 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:40.256808 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.258043 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.259906 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.260564 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.260694 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.261124 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:40.261770 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:40.261914 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.261976 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.262068 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 00:59:40.270090 2572 raft_consensus.cc:1261] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.270090 2434 raft_consensus.cc:1261] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.271104 2648 consensus_queue.cc:1035] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:40.271471 2642 consensus_queue.cc:1035] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:40.280189 2645 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: TxnStatusTablet state changed. Reason: New leader 32784c536d4e446ea59d2eaaff19b24f. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.280462 2645 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.282431 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298378875976384512 to 7298378875855032320. Current Snapshot: MvccSnapshot[applied={T|T < 7298378875976384512}]
I20260619 00:59:40.283546 2664 mvcc.cc:205] Tried to move back new op lower bound from 7298378875976384512 to 7298378875855032320. Current Snapshot: MvccSnapshot[applied={T|T < 7298378875976384512}]
I20260619 00:59:40.284314 2645 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.284605 2645 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P f8f4b1b6b5074610a2e3b95b953354b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.287287 2648 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.287557 2648 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:40.289680 2643 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: TxnStatusTablet state changed. Reason: New leader 32784c536d4e446ea59d2eaaff19b24f. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.289991 2643 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.292920 2642 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.293200 2642 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P 32784c536d4e446ea59d2eaaff19b24f: This TxnStatusTablet replica's current role is: LEADER
I20260619 00:59:40.295048 2643 tablet_replica.cc:439] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } } }
I20260619 00:59:40.295308 2643 tablet_replica.cc:442] T 161681821cf54205a547889a072db518 P bb7c979727bd4df29403bc42ae802e83: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 00:59:40.305523 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.299728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.0.97.1:33077)
0619 00:59:40.299905 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.299913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.300314 (+ 401us) server_negotiation.cc:407] Connection header received
0619 00:59:40.300533 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.300538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.300623 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.300721 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:40.301241 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.302192 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.303668 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.304083 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.304233 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.304680 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:40.304819 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:40.304925 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.305084 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.305180 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 00:59:40.323177 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.316785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35907 (local address 127.0.97.1:33077)
0619 00:59:40.317030 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:40.317037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:40.317408 (+ 371us) server_negotiation.cc:407] Connection header received
0619 00:59:40.317603 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:40.317608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:40.317670 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:40.317750 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:40.318356 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.319257 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.321340 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.321783 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.321897 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.322476 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:40.322602 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:40.322675 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:40.322854 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 00:59:40.322915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":71}
I20260619 00:59:40.323518 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:40.316719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33077 (local address 127.0.97.1:35907)
0619 00:59:40.317301 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:40.317321 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:40.317467 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:40.317795 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:40.317802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:40.317827 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:40.318201 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.318212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.319385 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.319391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.321185 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:40.321203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:40.321983 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:40.321989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:40.322189 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:40.322261 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:40.322705 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:40.322735 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 00:59:40.322839 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 00:59:40.322980 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":294,"threads_started":1}
I20260619 00:59:40.333038 2434 raft_consensus.cc:1261] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.333048 2572 raft_consensus.cc:1261] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.333618 2676 consensus_queue.cc:1035] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:40.333909 2648 consensus_queue.cc:1035] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:40.341156 2665 mvcc.cc:205] Tried to move back new op lower bound from 7298378876238274560 to 7298378875127271424. Current Snapshot: MvccSnapshot[applied={T|T < 7298378876238274560}]
I20260619 00:59:40.397380 2133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:40.397670 2133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:40.401067 2133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Sending DeleteTablet for 3 replicas of tablet c25c0dce71ec4998842ce597fd0a0a90
I20260619 00:59:40.402271 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260619 00:59:40.402578 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260619 00:59:40.403107 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet c25c0dce71ec4998842ce597fd0a0a90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:40 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41970
I20260619 00:59:40.404542 2133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:40.405405 2133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:40.413667 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 (DEFAULT_TABLE table=test-table [id=7783c66692da47c5b21dc04e4a64d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:40.413969 2547 tablet_service.cc:1467] Processing CreateTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 (DEFAULT_TABLE table=test-table [id=7783c66692da47c5b21dc04e4a64d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:40.414366 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b7fab3abe948e683bcc1a92308a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.414439 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 (DEFAULT_TABLE table=test-table [id=7783c66692da47c5b21dc04e4a64d0c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:40.414595 2547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b7fab3abe948e683bcc1a92308a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.414855 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35b7fab3abe948e683bcc1a92308a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:40.420964 2638 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Bootstrap starting.
I20260619 00:59:40.422598 2638 tablet_bootstrap.cc:654] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.422942 2637 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Bootstrap starting.
I20260619 00:59:40.423831 2682 tablet_replica.cc:330] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: stopping tablet replica
I20260619 00:59:40.424429 2683 tablet_replica.cc:330] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: stopping tablet replica
I20260619 00:59:40.424966 2637 tablet_bootstrap.cc:654] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.425414 2682 raft_consensus.cc:2229] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:40.425848 2682 raft_consensus.cc:2258] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:40.426927 2641 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Bootstrap starting.
I20260619 00:59:40.428243 2682 ts_tablet_manager.cc:1905] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:40.428385 2637 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: No bootstrap required, opened a new log
I20260619 00:59:40.428442 2641 tablet_bootstrap.cc:654] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:40.428529 2637 ts_tablet_manager.cc:1397] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260619 00:59:40.429584 2637 raft_consensus.cc:348] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.430128 2637 raft_consensus.cc:374] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.430224 2637 raft_consensus.cc:729] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.430760 2641 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: No bootstrap required, opened a new log
I20260619 00:59:40.430573 2637 consensus_queue.cc:260] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.430927 2641 ts_tablet_manager.cc:1397] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260619 00:59:40.431284 2637 ts_tablet_manager.cc:1428] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:40.431625 2641 raft_consensus.cc:348] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.431855 2641 raft_consensus.cc:374] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.431923 2641 raft_consensus.cc:729] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4b1b6b5074610a2e3b95b953354b8, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.432128 2641 consensus_queue.cc:260] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.432614 2641 ts_tablet_manager.cc:1428] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:59:40.424995 2683 raft_consensus.cc:2229] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:40.436123 2683 raft_consensus.cc:2258] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:40.436832 2681 tablet_replica.cc:330] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: stopping tablet replica
I20260619 00:59:40.437253 2681 raft_consensus.cc:2229] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:40.437594 2681 raft_consensus.cc:2258] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:40.438083 2683 ts_tablet_manager.cc:1905] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:40.439260 2681 ts_tablet_manager.cc:1905] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:40.440485 2638 tablet_bootstrap.cc:492] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: No bootstrap required, opened a new log
I20260619 00:59:40.440739 2638 ts_tablet_manager.cc:1397] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20260619 00:59:40.441556 2638 raft_consensus.cc:348] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.441922 2638 raft_consensus.cc:374] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:40.442037 2638 raft_consensus.cc:729] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Initialized, Role: FOLLOWER
I20260619 00:59:40.442389 2638 consensus_queue.cc:260] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.443121 2638 ts_tablet_manager.cc:1428] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:59:40.446307 2681 ts_tablet_manager.cc:1918] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:40.446533 2681 log.cc:1196] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/c25c0dce71ec4998842ce597fd0a0a90
I20260619 00:59:40.447060 2682 ts_tablet_manager.cc:1918] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:40.447120 2681 ts_tablet_manager.cc:1939] T c25c0dce71ec4998842ce597fd0a0a90 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting consensus metadata
I20260619 00:59:40.447337 2682 log.cc:1196] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/c25c0dce71ec4998842ce597fd0a0a90
I20260619 00:59:40.447888 2682 ts_tablet_manager.cc:1939] T c25c0dce71ec4998842ce597fd0a0a90 P bb7c979727bd4df29403bc42ae802e83: Deleting consensus metadata
I20260619 00:59:40.448963 2117 catalog_manager.cc:4841] TS f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607): tablet c25c0dce71ec4998842ce597fd0a0a90 (table test-table [id=75bba46bf63c47f6b58d6bf35ed73228]) successfully deleted
I20260619 00:59:40.450044 2683 ts_tablet_manager.cc:1918] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:40.450230 2683 log.cc:1196] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/c25c0dce71ec4998842ce597fd0a0a90
I20260619 00:59:40.450846 2683 ts_tablet_manager.cc:1939] T c25c0dce71ec4998842ce597fd0a0a90 P 32784c536d4e446ea59d2eaaff19b24f: Deleting consensus metadata
I20260619 00:59:40.450815 2118 catalog_manager.cc:4841] TS bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149): tablet c25c0dce71ec4998842ce597fd0a0a90 (table test-table [id=75bba46bf63c47f6b58d6bf35ed73228]) successfully deleted
I20260619 00:59:40.452168 2120 catalog_manager.cc:4841] TS 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077): tablet c25c0dce71ec4998842ce597fd0a0a90 (table test-table [id=75bba46bf63c47f6b58d6bf35ed73228]) successfully deleted
I20260619 00:59:40.553567 2676 raft_consensus.cc:482] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:40.553815 2676 raft_consensus.cc:504] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.554560 2676 leader_election.cc:288] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:40.555173 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b7fab3abe948e683bcc1a92308a1b0" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83" is_pre_election: true
I20260619 00:59:40.555423 2434 raft_consensus.cc:2454] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.555466 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b7fab3abe948e683bcc1a92308a1b0" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" is_pre_election: true
I20260619 00:59:40.555708 2572 raft_consensus.cc:2454] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:40.555920 2221 leader_election.cc:302] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83; no voters:
I20260619 00:59:40.556238 2676 raft_consensus.cc:2781] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:40.556342 2676 raft_consensus.cc:482] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:40.556490 2676 raft_consensus.cc:3037] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.558946 2676 raft_consensus.cc:504] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.559590 2676 leader_election.cc:288] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Requested vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:40.560066 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b7fab3abe948e683bcc1a92308a1b0" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
I20260619 00:59:40.560120 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35b7fab3abe948e683bcc1a92308a1b0" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83"
I20260619 00:59:40.560266 2572 raft_consensus.cc:3037] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.560300 2434 raft_consensus.cc:3037] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:40.563184 2572 raft_consensus.cc:2454] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.563184 2434 raft_consensus.cc:2454] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:40.563596 2220 leader_election.cc:302] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, f8f4b1b6b5074610a2e3b95b953354b8; no voters:
I20260619 00:59:40.563874 2676 raft_consensus.cc:2781] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:40.564065 2676 raft_consensus.cc:686] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Becoming Leader. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Running, Role: LEADER
I20260619 00:59:40.564322 2676 consensus_queue.cc:237] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:40.567202 2131 catalog_manager.cc:5495] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } health_report { overall_health: UNKNOWN } } }
I20260619 00:59:40.606904 2433 raft_consensus.cc:1261] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.607247 2572 raft_consensus.cc:1261] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:40.607573 2648 consensus_queue.cc:1035] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:40.607896 2676 consensus_queue.cc:1035] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 00:59:40.636385 2663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 00:59:42.229027 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:42.229274 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:42.233219 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Sending DeleteTablet for 3 replicas of tablet 35b7fab3abe948e683bcc1a92308a1b0
I20260619 00:59:42.234066 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260619 00:59:42.234140 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260619 00:59:42.234489 2272 tablet_service.cc:1514] Processing DeleteTablet for tablet 35b7fab3abe948e683bcc1a92308a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41970
I20260619 00:59:42.234699 2711 tablet_replica.cc:330] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: stopping tablet replica
I20260619 00:59:42.235016 2711 raft_consensus.cc:2229] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:42.235836 2711 raft_consensus.cc:2258] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.243495 2711 ts_tablet_manager.cc:1905] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.246436 2713 tablet_replica.cc:330] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: stopping tablet replica
I20260619 00:59:42.246577 2712 tablet_replica.cc:330] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: stopping tablet replica
I20260619 00:59:42.246747 2713 raft_consensus.cc:2229] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:42.246827 2712 raft_consensus.cc:2229] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:42.247133 2712 raft_consensus.cc:2258] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.247131 2713 raft_consensus.cc:2258] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.248078 2713 ts_tablet_manager.cc:1905] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.248152 2712 ts_tablet_manager.cc:1905] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.251246 2711 ts_tablet_manager.cc:1918] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:42.251439 2711 log.cc:1196] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/35b7fab3abe948e683bcc1a92308a1b0
I20260619 00:59:42.251971 2711 ts_tablet_manager.cc:1939] T 35b7fab3abe948e683bcc1a92308a1b0 P bb7c979727bd4df29403bc42ae802e83: Deleting consensus metadata
I20260619 00:59:42.253288 2118 catalog_manager.cc:4841] TS bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149): tablet 35b7fab3abe948e683bcc1a92308a1b0 (table test-table [id=7783c66692da47c5b21dc04e4a64d0c7]) successfully deleted
I20260619 00:59:42.254626 2713 ts_tablet_manager.cc:1918] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:42.254784 2712 ts_tablet_manager.cc:1918] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:42.254837 2713 log.cc:1196] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/35b7fab3abe948e683bcc1a92308a1b0
I20260619 00:59:42.254949 2712 log.cc:1196] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/35b7fab3abe948e683bcc1a92308a1b0
I20260619 00:59:42.255318 2713 ts_tablet_manager.cc:1939] T 35b7fab3abe948e683bcc1a92308a1b0 P 32784c536d4e446ea59d2eaaff19b24f: Deleting consensus metadata
I20260619 00:59:42.255381 2712 ts_tablet_manager.cc:1939] T 35b7fab3abe948e683bcc1a92308a1b0 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting consensus metadata
I20260619 00:59:42.256371 2120 catalog_manager.cc:4841] TS 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077): tablet 35b7fab3abe948e683bcc1a92308a1b0 (table test-table [id=7783c66692da47c5b21dc04e4a64d0c7]) successfully deleted
I20260619 00:59:42.256510 2117 catalog_manager.cc:4841] TS f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607): tablet 35b7fab3abe948e683bcc1a92308a1b0 (table test-table [id=7783c66692da47c5b21dc04e4a64d0c7]) successfully deleted
Jun 19 00:59:42 dist-test-slave-3rxd krb5kdc[2088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 00:59:42 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:42.286939 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.271697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.0.97.62:45499)
0619 00:59:42.272335 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.272344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.272447 (+ 103us) server_negotiation.cc:407] Connection header received
0619 00:59:42.272702 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.272707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.272792 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.272929 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:42.274286 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.275180 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.276516 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.276929 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.277080 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.281508 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:42.281514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:42.281517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:42.281552 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:42.284171 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:42.284854 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:42.284862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:42.284868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:42.285021 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:42.285412 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:42.285420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:42.285423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:42.285699 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:42.285962 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.286311 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.286548 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":302,"threads_started":1}
I20260619 00:59:42.290216 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:42.290885 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:42.296854 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 99c6402257d24187a9c6384eebfc2d70 (DEFAULT_TABLE table=test-table [id=d5026c1f6cb646f7a38772a087082c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.296947 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 99c6402257d24187a9c6384eebfc2d70 (DEFAULT_TABLE table=test-table [id=d5026c1f6cb646f7a38772a087082c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.297091 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 99c6402257d24187a9c6384eebfc2d70 (DEFAULT_TABLE table=test-table [id=d5026c1f6cb646f7a38772a087082c33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.297277 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c6402257d24187a9c6384eebfc2d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.297349 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c6402257d24187a9c6384eebfc2d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.297443 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99c6402257d24187a9c6384eebfc2d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.302559 2722 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Bootstrap starting.
I20260619 00:59:42.304262 2722 tablet_bootstrap.cc:654] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.306077 2722 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: No bootstrap required, opened a new log
I20260619 00:59:42.306283 2722 ts_tablet_manager.cc:1397] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 00:59:42.307145 2722 raft_consensus.cc:348] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.307344 2722 raft_consensus.cc:374] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.307443 2722 raft_consensus.cc:729] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.307687 2722 consensus_queue.cc:260] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.308211 2722 ts_tablet_manager.cc:1428] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 00:59:42.312191 2723 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Bootstrap starting.
I20260619 00:59:42.312739 2724 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Bootstrap starting.
I20260619 00:59:42.314913 2724 tablet_bootstrap.cc:654] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.314914 2723 tablet_bootstrap.cc:654] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.316891 2723 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: No bootstrap required, opened a new log
I20260619 00:59:42.317061 2723 ts_tablet_manager.cc:1397] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 00:59:42.317553 2724 tablet_bootstrap.cc:492] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: No bootstrap required, opened a new log
I20260619 00:59:42.317687 2724 ts_tablet_manager.cc:1397] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 00:59:42.317821 2723 raft_consensus.cc:348] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.318070 2723 raft_consensus.cc:374] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.318176 2723 raft_consensus.cc:729] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.318449 2723 consensus_queue.cc:260] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.318473 2724 raft_consensus.cc:348] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.318686 2724 raft_consensus.cc:374] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.318812 2724 raft_consensus.cc:729] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4b1b6b5074610a2e3b95b953354b8, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.319133 2724 consensus_queue.cc:260] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.319370 2723 ts_tablet_manager.cc:1428] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 00:59:42.320384 2724 ts_tablet_manager.cc:1428] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:42.421252 2706 raft_consensus.cc:482] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:42.421491 2706 raft_consensus.cc:504] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.422099 2706 leader_election.cc:288] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:42.422785 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c6402257d24187a9c6384eebfc2d70" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83" is_pre_election: true
I20260619 00:59:42.422818 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c6402257d24187a9c6384eebfc2d70" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" is_pre_election: true
I20260619 00:59:42.423136 2433 raft_consensus.cc:2454] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:42.423178 2572 raft_consensus.cc:2454] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 0.
I20260619 00:59:42.423687 2221 leader_election.cc:302] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83; no voters:
I20260619 00:59:42.423979 2706 raft_consensus.cc:2781] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:42.424100 2706 raft_consensus.cc:482] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:42.424192 2706 raft_consensus.cc:3037] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.426296 2706 raft_consensus.cc:504] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.426867 2706 leader_election.cc:288] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Requested vote from peers bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:42.427394 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c6402257d24187a9c6384eebfc2d70" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb7c979727bd4df29403bc42ae802e83"
I20260619 00:59:42.427431 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99c6402257d24187a9c6384eebfc2d70" candidate_uuid: "32784c536d4e446ea59d2eaaff19b24f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
I20260619 00:59:42.427610 2572 raft_consensus.cc:3037] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.427627 2433 raft_consensus.cc:3037] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.430567 2572 raft_consensus.cc:2454] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:42.430550 2433 raft_consensus.cc:2454] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32784c536d4e446ea59d2eaaff19b24f in term 1.
I20260619 00:59:42.431037 2221 leader_election.cc:302] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83, f8f4b1b6b5074610a2e3b95b953354b8; no voters:
I20260619 00:59:42.431337 2706 raft_consensus.cc:2781] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:42.431469 2706 raft_consensus.cc:686] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Becoming Leader. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Running, Role: LEADER
I20260619 00:59:42.431731 2706 consensus_queue.cc:237] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.434154 2132 catalog_manager.cc:5495] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f reported cstate change: term changed from 0 to 1, leader changed from <none> to 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "32784c536d4e446ea59d2eaaff19b24f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } health_report { overall_health: UNKNOWN } } }
I20260619 00:59:42.471221 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.465893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52186 (local address 127.0.97.1:33077)
0619 00:59:42.466862 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.466867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.466893 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:59:42.466947 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.466950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.467007 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.467089 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:42.467401 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.468159 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.469391 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.469914 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.470011 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.470494 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.470586 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.470654 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.470806 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.470861 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":318,"threads_started":1}
I20260619 00:59:42.479511 2433 raft_consensus.cc:1261] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:42.479547 2571 raft_consensus.cc:1261] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Refusing update from remote peer 32784c536d4e446ea59d2eaaff19b24f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:42.480074 2676 consensus_queue.cc:1035] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:42.480331 2706 consensus_queue.cc:1035] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:42.520119 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:42.520329 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:42.522825 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Sending DeleteTablet for 3 replicas of tablet 99c6402257d24187a9c6384eebfc2d70
I20260619 00:59:42.523453 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c6402257d24187a9c6384eebfc2d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41970
I20260619 00:59:42.523520 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c6402257d24187a9c6384eebfc2d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260619 00:59:42.523592 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 99c6402257d24187a9c6384eebfc2d70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260619 00:59:42.523744 2713 tablet_replica.cc:330] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: stopping tablet replica
I20260619 00:59:42.523738 2711 tablet_replica.cc:330] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: stopping tablet replica
I20260619 00:59:42.523880 2712 tablet_replica.cc:330] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: stopping tablet replica
I20260619 00:59:42.523926 2711 raft_consensus.cc:2229] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:42.523957 2713 raft_consensus.cc:2229] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:42.524091 2712 raft_consensus.cc:2229] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:42.524107 2711 raft_consensus.cc:2258] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.524220 2713 raft_consensus.cc:2258] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.524283 2712 raft_consensus.cc:2258] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:42.525301 2711 ts_tablet_manager.cc:1905] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.525460 2712 ts_tablet_manager.cc:1905] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.525746 2713 ts_tablet_manager.cc:1905] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:42.529302 2132 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 00:59:42.530277 2132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 00:59:42.531391 2712 ts_tablet_manager.cc:1918] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:42.531528 2712 log.cc:1196] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/99c6402257d24187a9c6384eebfc2d70
I20260619 00:59:42.531810 2711 ts_tablet_manager.cc:1918] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:42.531930 2712 ts_tablet_manager.cc:1939] T 99c6402257d24187a9c6384eebfc2d70 P f8f4b1b6b5074610a2e3b95b953354b8: Deleting consensus metadata
I20260619 00:59:42.531955 2711 log.cc:1196] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/99c6402257d24187a9c6384eebfc2d70
I20260619 00:59:42.532024 2713 ts_tablet_manager.cc:1918] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 00:59:42.532151 2713 log.cc:1196] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/99c6402257d24187a9c6384eebfc2d70
I20260619 00:59:42.532522 2713 ts_tablet_manager.cc:1939] T 99c6402257d24187a9c6384eebfc2d70 P 32784c536d4e446ea59d2eaaff19b24f: Deleting consensus metadata
I20260619 00:59:42.532902 2117 catalog_manager.cc:4841] TS f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607): tablet 99c6402257d24187a9c6384eebfc2d70 (table test-table [id=d5026c1f6cb646f7a38772a087082c33]) successfully deleted
I20260619 00:59:42.533176 2711 ts_tablet_manager.cc:1939] T 99c6402257d24187a9c6384eebfc2d70 P bb7c979727bd4df29403bc42ae802e83: Deleting consensus metadata
I20260619 00:59:42.534090 2120 catalog_manager.cc:4841] TS 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077): tablet 99c6402257d24187a9c6384eebfc2d70 (table test-table [id=d5026c1f6cb646f7a38772a087082c33]) successfully deleted
I20260619 00:59:42.534147 2118 catalog_manager.cc:4841] TS bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149): tablet 99c6402257d24187a9c6384eebfc2d70 (table test-table [id=d5026c1f6cb646f7a38772a087082c33]) successfully deleted
I20260619 00:59:42.537511 2269 tablet_service.cc:1467] Processing CreateTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (DEFAULT_TABLE table=test-table [id=0db041194a994838b9d94c4fc4e8c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.537616 2409 tablet_service.cc:1467] Processing CreateTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (DEFAULT_TABLE table=test-table [id=0db041194a994838b9d94c4fc4e8c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.537914 2269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.537842 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (DEFAULT_TABLE table=test-table [id=0db041194a994838b9d94c4fc4e8c1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 00:59:42.537992 2409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.538233 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:42.541996 2722 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Bootstrap starting.
I20260619 00:59:42.543856 2722 tablet_bootstrap.cc:654] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.544209 2723 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Bootstrap starting.
I20260619 00:59:42.545145 2724 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Bootstrap starting.
I20260619 00:59:42.545753 2723 tablet_bootstrap.cc:654] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.546792 2724 tablet_bootstrap.cc:654] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:42.547742 2723 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: No bootstrap required, opened a new log
I20260619 00:59:42.547900 2723 ts_tablet_manager.cc:1397] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260619 00:59:42.548491 2722 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: No bootstrap required, opened a new log
I20260619 00:59:42.548641 2722 ts_tablet_manager.cc:1397] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 00:59:42.548620 2723 raft_consensus.cc:348] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.548877 2723 raft_consensus.cc:374] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.548954 2723 raft_consensus.cc:729] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.549177 2723 consensus_queue.cc:260] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.549348 2722 raft_consensus.cc:348] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.549585 2722 raft_consensus.cc:374] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.549669 2722 raft_consensus.cc:729] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32784c536d4e446ea59d2eaaff19b24f, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.549878 2722 consensus_queue.cc:260] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.550216 2723 ts_tablet_manager.cc:1428] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:59:42.550356 2722 ts_tablet_manager.cc:1428] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 00:59:42.552079 2724 tablet_bootstrap.cc:492] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: No bootstrap required, opened a new log
I20260619 00:59:42.552229 2724 ts_tablet_manager.cc:1397] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260619 00:59:42.552927 2724 raft_consensus.cc:348] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.553190 2724 raft_consensus.cc:374] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:42.553297 2724 raft_consensus.cc:729] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8f4b1b6b5074610a2e3b95b953354b8, State: Initialized, Role: FOLLOWER
I20260619 00:59:42.553567 2724 consensus_queue.cc:260] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.554076 2724 ts_tablet_manager.cc:1428] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 00:59:42.601754 2728 raft_consensus.cc:482] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 00:59:42.602012 2728 raft_consensus.cc:504] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.603472 2728 leader_election.cc:288] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:42.610781 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.603839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41127 (local address 127.0.97.1:33077)
0619 00:59:42.604065 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.604074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.604789 (+ 715us) server_negotiation.cc:407] Connection header received
0619 00:59:42.605121 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.605125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.605184 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.605279 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:42.605798 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.606505 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.609101 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.609864 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.610011 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.610118 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.610383 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.610513 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 00:59:42.610926 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.603751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33077 (local address 127.0.97.2:41127)
0619 00:59:42.604647 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:42.604680 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:42.604926 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:42.605304 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:42.605308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:42.605330 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:42.605620 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.605628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.606643 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.606650 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.608942 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.608956 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.610060 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.610066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.610224 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.610247 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 00:59:42.610361 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 00:59:42.610491 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":88,"thread_start_us":912,"threads_started":1}
I20260619 00:59:42.611193 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.603839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48475 (local address 127.0.97.3:38607)
0619 00:59:42.605206 (+ 1367us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.605214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.605238 (+ 24us) server_negotiation.cc:407] Connection header received
0619 00:59:42.605300 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.605304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.605362 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.605470 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 00:59:42.606075 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.606869 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.609367 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.610176 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.610343 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.610468 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.610752 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.610881 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1239,"thread_start_us":1104,"threads_started":1}
I20260619 00:59:42.611230 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.603788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38607 (local address 127.0.97.2:48475)
0619 00:59:42.604990 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:42.605008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:42.605124 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:42.605490 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:42.605494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:42.605523 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 00:59:42.605918 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.605926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.606994 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.607002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.609217 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.609228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.610398 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.610404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.610567 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.610589 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 00:59:42.610691 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 00:59:42.610844 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":24,"thread_start_us":437,"threads_started":1}
I20260619 00:59:42.611951 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6615c6b88a7649b3b0b5ed2f2fa3e56f" candidate_uuid: "bb7c979727bd4df29403bc42ae802e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" is_pre_election: true
I20260619 00:59:42.611927 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6615c6b88a7649b3b0b5ed2f2fa3e56f" candidate_uuid: "bb7c979727bd4df29403bc42ae802e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32784c536d4e446ea59d2eaaff19b24f" is_pre_election: true
I20260619 00:59:42.612282 2571 raft_consensus.cc:2454] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb7c979727bd4df29403bc42ae802e83 in term 0.
I20260619 00:59:42.612346 2295 raft_consensus.cc:2454] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb7c979727bd4df29403bc42ae802e83 in term 0.
I20260619 00:59:42.612759 2360 leader_election.cc:302] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb7c979727bd4df29403bc42ae802e83, f8f4b1b6b5074610a2e3b95b953354b8; no voters:
I20260619 00:59:42.613137 2728 raft_consensus.cc:2781] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 00:59:42.613260 2728 raft_consensus.cc:482] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 00:59:42.613368 2728 raft_consensus.cc:3037] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.615480 2728 raft_consensus.cc:504] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.616083 2728 leader_election.cc:288] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [CANDIDATE]: Term 1 election: Requested vote from peers 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077), f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607)
I20260619 00:59:42.616448 2295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6615c6b88a7649b3b0b5ed2f2fa3e56f" candidate_uuid: "bb7c979727bd4df29403bc42ae802e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32784c536d4e446ea59d2eaaff19b24f"
I20260619 00:59:42.616576 2571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6615c6b88a7649b3b0b5ed2f2fa3e56f" candidate_uuid: "bb7c979727bd4df29403bc42ae802e83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8f4b1b6b5074610a2e3b95b953354b8"
I20260619 00:59:42.616654 2295 raft_consensus.cc:3037] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.616789 2571 raft_consensus.cc:3037] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:42.619638 2571 raft_consensus.cc:2454] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7c979727bd4df29403bc42ae802e83 in term 1.
I20260619 00:59:42.619656 2295 raft_consensus.cc:2454] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb7c979727bd4df29403bc42ae802e83 in term 1.
I20260619 00:59:42.620035 2363 leader_election.cc:302] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32784c536d4e446ea59d2eaaff19b24f, bb7c979727bd4df29403bc42ae802e83; no voters:
I20260619 00:59:42.620316 2728 raft_consensus.cc:2781] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:42.620710 2728 raft_consensus.cc:686] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 1 LEADER]: Becoming Leader. State: Replica: bb7c979727bd4df29403bc42ae802e83, State: Running, Role: LEADER
I20260619 00:59:42.621059 2728 consensus_queue.cc:237] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } }
I20260619 00:59:42.624378 2131 catalog_manager.cc:5495] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb7c979727bd4df29403bc42ae802e83 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "bb7c979727bd4df29403bc42ae802e83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb7c979727bd4df29403bc42ae802e83" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 } health_report { overall_health: UNKNOWN } } }
I20260619 00:59:42.662674 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.656528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.0.97.2:44149)
0619 00:59:42.657557 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.657561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.657577 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:59:42.657657 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.657660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.657731 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.657909 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:42.658310 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.659194 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.660468 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.660883 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.661019 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.661471 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.662045 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.662205 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.662283 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.662357 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":939,"thread_start_us":814,"threads_started":1}
I20260619 00:59:42.677924 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.672131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56393 (local address 127.0.97.1:33077)
0619 00:59:42.672331 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.672337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.672659 (+ 322us) server_negotiation.cc:407] Connection header received
0619 00:59:42.672830 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.672836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.672900 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.672974 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:42.673573 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.674425 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.676377 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.676770 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.676867 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.677360 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.677454 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.677519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.677661 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.677723 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 00:59:42.678128 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.671947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33077 (local address 127.0.97.2:56393)
0619 00:59:42.672561 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:42.672581 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:42.672716 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:42.673046 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:42.673050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:42.673059 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:42.673417 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.673426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.674553 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.674559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.676236 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.676249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.676930 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.676937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.677131 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.677193 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.677545 (+ 352us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.677560 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 00:59:42.677636 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 00:59:42.677766 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":274,"threads_started":1}
I20260619 00:59:42.686084 2571 raft_consensus.cc:1261] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Refusing update from remote peer bb7c979727bd4df29403bc42ae802e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:42.686326 2295 raft_consensus.cc:1261] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Refusing update from remote peer bb7c979727bd4df29403bc42ae802e83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 00:59:42.686868 2742 consensus_queue.cc:1035] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8f4b1b6b5074610a2e3b95b953354b8" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:42.687224 2728 consensus_queue.cc:1035] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32784c536d4e446ea59d2eaaff19b24f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 00:59:42.714105 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.708387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47197 (local address 127.0.97.2:44149)
0619 00:59:42.708544 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:42.708550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:42.709236 (+ 686us) server_negotiation.cc:407] Connection header received
0619 00:59:42.709374 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:42.709378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:42.709434 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:42.709503 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 00:59:42.709941 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.710773 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.712412 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.712885 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.713008 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.713376 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.713500 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.713582 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:42.713729 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 00:59:42.713800 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 00:59:42.714174 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:42.708436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44149 (local address 127.0.97.1:47197)
0619 00:59:42.709130 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:42.709155 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:42.709275 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:42.709530 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:42.709533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:42.709542 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 00:59:42.709816 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.709823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.710911 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.710918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.712225 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:42.712236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:42.713062 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:42.713070 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:42.713188 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:42.713222 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 00:59:42.713582 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 00:59:42.713595 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 00:59:42.713709 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 00:59:42.713835 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":401,"threads_started":1}
W20260619 00:59:42.715286 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 00:59:44.313395 2132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:44.313628 2132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 00:59:44.317320 2132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24d33cf6934b4c7598285befc368eafe: Sending DeleteTablet for 3 replicas of tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f
I20260619 00:59:44.318084 2269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41970
I20260619 00:59:44.318148 2409 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260619 00:59:44.318193 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 00:59:44 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36172
I20260619 00:59:44.319064 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2204
I20260619 00:59:44.328239 2779 tablet_replica.cc:330] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: stopping tablet replica
I20260619 00:59:44.328881 2779 raft_consensus.cc:2229] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:44.329265 2779 raft_consensus.cc:2258] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:44.330770 2779 ts_tablet_manager.cc:1905] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:44.339172 2778 tablet_replica.cc:330] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: stopping tablet replica
I20260619 00:59:44.339180 2777 tablet_replica.cc:330] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: stopping tablet replica
I20260619 00:59:44.339395 2778 raft_consensus.cc:2229] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 1 LEADER]: Raft consensus shutting down.
I20260619 00:59:44.339458 2777 raft_consensus.cc:2229] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 00:59:44.339630 2779 ts_tablet_manager.cc:1918] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:44.339797 2779 log.cc:1196] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/6615c6b88a7649b3b0b5ed2f2fa3e56f
I20260619 00:59:44.339946 2777 raft_consensus.cc:2258] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:44.340042 2778 raft_consensus.cc:2258] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 00:59:44.340296 2779 ts_tablet_manager.cc:1939] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P f8f4b1b6b5074610a2e3b95b953354b8: Deleting consensus metadata
I20260619 00:59:44.341188 2777 ts_tablet_manager.cc:1905] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:44.341490 2778 ts_tablet_manager.cc:1905] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 00:59:44.341564 2117 catalog_manager.cc:4841] TS f8f4b1b6b5074610a2e3b95b953354b8 (127.0.97.3:38607): tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (table test-table [id=0db041194a994838b9d94c4fc4e8c1b9]) successfully deleted
I20260619 00:59:44.347357 2777 ts_tablet_manager.cc:1918] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:44.347520 2777 log.cc:1196] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/6615c6b88a7649b3b0b5ed2f2fa3e56f
I20260619 00:59:44.347590 2778 ts_tablet_manager.cc:1918] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 00:59:44.347720 2778 log.cc:1196] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/6615c6b88a7649b3b0b5ed2f2fa3e56f
I20260619 00:59:44.348057 2777 ts_tablet_manager.cc:1939] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P 32784c536d4e446ea59d2eaaff19b24f: Deleting consensus metadata
I20260619 00:59:44.348183 2778 ts_tablet_manager.cc:1939] T 6615c6b88a7649b3b0b5ed2f2fa3e56f P bb7c979727bd4df29403bc42ae802e83: Deleting consensus metadata
I20260619 00:59:44.348999 2120 catalog_manager.cc:4841] TS 32784c536d4e446ea59d2eaaff19b24f (127.0.97.1:33077): tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (table test-table [id=0db041194a994838b9d94c4fc4e8c1b9]) successfully deleted
I20260619 00:59:44.349061 2118 catalog_manager.cc:4841] TS bb7c979727bd4df29403bc42ae802e83 (127.0.97.2:44149): tablet 6615c6b88a7649b3b0b5ed2f2fa3e56f (table test-table [id=0db041194a994838b9d94c4fc4e8c1b9]) successfully deleted
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:44.361689 2781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:44.320918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52202 (local address 127.0.97.1:33077)
0619 00:59:44.340715 (+ 19797us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:44.340722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:44.340747 (+ 25us) server_negotiation.cc:407] Connection header received
0619 00:59:44.340809 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:44.340814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:44.340897 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:44.341022 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:44.342860 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.344127 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.345878 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.346596 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.346765 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:44.356178 (+ 9413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:44.356192 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:44.356208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:44.356244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:44.359031 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.359656 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.359663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.359665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.359808 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.360162 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.360171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.360174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.360538 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:44.360818 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:44.361113 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 00:59:44.361320 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19674,"thread_start_us":20364,"threads_started":1}
W20260619 00:59:44.362296 2335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52202
W20260619 00:59:44.362957 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:44.363168 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2204
I20260619 00:59:44.388281 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2344
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:44.402334 2782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:44.388986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.0.97.2:44149)
0619 00:59:44.389502 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:44.389506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:44.389522 (+ 16us) server_negotiation.cc:407] Connection header received
0619 00:59:44.389586 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:44.389590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:44.389647 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:44.389742 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:44.391059 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.391761 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.392970 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.393358 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.393450 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:44.397299 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:44.397311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:44.397326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:44.397354 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:44.399926 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.400503 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.400509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.400511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.400625 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.401005 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.401008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.401010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.401401 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:44.401618 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:44.401838 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 00:59:44.401996 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":293,"threads_started":1}
W20260619 00:59:44.403178 2474 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47732
W20260619 00:59:44.403800 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2344: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:44.403937 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2344
I20260619 00:59:44.425266 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2483
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:44.440167 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:44.426075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.0.97.3:38607)
0619 00:59:44.426602 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:44.426607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:44.426624 (+ 17us) server_negotiation.cc:407] Connection header received
0619 00:59:44.426675 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:44.426679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:44.426807 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:44.426927 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:44.428193 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.429107 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.430495 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:44.431071 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:44.431201 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:44.435263 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:44.435276 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:44.435293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:44.435323 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:44.437953 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.438530 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.438536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.438538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.438660 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:44.439031 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:44.439034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:44.439036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:44.439337 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:44.439557 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:44.439767 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 00:59:44.439919 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":259,"threads_started":1}
W20260619 00:59:44.440848 2613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49214
W20260619 00:59:44.441426 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:44.441601 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2483
I20260619 00:59:44.459033 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2101
I20260619 00:59:44.460173 2173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:44.574272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2101
2026-06-19T00:59:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8262 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2787](info): set up 2 sockets
Jun 19 00:59:44 dist-test-slave-3rxd krb5kdc[2787](info): commencing operation
krb5kdc: starting...
W20260619 00:59:45.651022 388 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:45 dist-test-slave-3rxd krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:45.834231 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36873
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:46.149030 2800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:46.149300 2800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:46.149472 2800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:46.156596 2800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:46.156699 2800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:46.156816 2800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:46.156911 2800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:46.156981 2800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:46.166479 2800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:46.167752 2800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:46.169062 2800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:46.180474 2808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:46.180518 2806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:46.180542 2805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:46.181125 2800 server_base.cc:1034] running on GCE node
I20260619 00:59:46.181906 2800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:46.183302 2800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:46.184525 2800 hybrid_clock.cc:648] HybridClock initialized: now 1781830786184476 us; error 58 us; skew 500 ppm
I20260619 00:59:46.184862 2800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:46 dist-test-slave-3rxd krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:46.187863 2800 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:46.189314 2800 webserver.cc:460] Webserver started at http://127.0.97.62:34649/ using document root <none> and password file <none>
I20260619 00:59:46.190004 2800 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:46.190145 2800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:46.190454 2800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:46.193414 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "077e0e8bf32048e3b319a560a916230e"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "bc23d41b2818b8d53b9960b5fabbbf33"
server_key_iv: "8556bc768c47cb4c99cafe378727d161"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.194195 2800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "077e0e8bf32048e3b319a560a916230e"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "bc23d41b2818b8d53b9960b5fabbbf33"
server_key_iv: "8556bc768c47cb4c99cafe378727d161"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.199255 2800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 00:59:46.202615 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:46.203981 2800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:59:46.204154 2800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "077e0e8bf32048e3b319a560a916230e"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "bc23d41b2818b8d53b9960b5fabbbf33"
server_key_iv: "8556bc768c47cb4c99cafe378727d161"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.204351 2800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:46.215574 2800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:46.219374 2800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:46.219655 2800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:46.237103 2800 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36873
I20260619 00:59:46.237133 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36873 every 8 connection(s)
I20260619 00:59:46.238458 2800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:46.240370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2800
I20260619 00:59:46.240588 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:46.241384 388 external_mini_cluster.cc:1428] Setting key 9609fe31023292ff11b34a9fd0919519
I20260619 00:59:46.242846 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 00:59:46 dist-test-slave-3rxd krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:46.256318 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e: Bootstrap starting.
I20260619 00:59:46.259611 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:46.260594 2867 log.cc:824] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:46.263305 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:46.243403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44146 (local address 127.0.97.62:36873)
0619 00:59:46.244155 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:46.244166 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:46.244204 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:59:46.245688 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:46.245707 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:46.246052 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:46.246389 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:46.248178 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.249457 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.251245 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.251869 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.251993 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:46.256329 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:46.256336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:46.256349 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:46.256381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:46.259381 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:46.260118 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:46.260124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:46.260130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:46.260280 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:46.260655 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:46.260664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:46.260667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:46.261343 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:46.261648 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:46.261960 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 00:59:46.262310 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":352,"threads_started":1}
I20260619 00:59:46.263535 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e: No bootstrap required, opened a new log
I20260619 00:59:46.270401 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } }
I20260619 00:59:46.270772 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:46.270886 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077e0e8bf32048e3b319a560a916230e, State: Initialized, Role: FOLLOWER
I20260619 00:59:46.271378 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } }
I20260619 00:59:46.271613 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:46.271745 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:46.271900 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:46.273967 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } }
I20260619 00:59:46.274411 2867 leader_election.cc:302] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077e0e8bf32048e3b319a560a916230e; no voters:
I20260619 00:59:46.274998 2867 leader_election.cc:288] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:46.275121 2872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:46.275749 2872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [term 1 LEADER]: Becoming Leader. State: Replica: 077e0e8bf32048e3b319a560a916230e, State: Running, Role: LEADER
I20260619 00:59:46.276207 2872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } }
I20260619 00:59:46.276857 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:46.278966 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077e0e8bf32048e3b319a560a916230e. Latest consensus state: current_term: 1 leader_uuid: "077e0e8bf32048e3b319a560a916230e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } } }
I20260619 00:59:46.279408 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:46.280148 2873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077e0e8bf32048e3b319a560a916230e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077e0e8bf32048e3b319a560a916230e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36873 } } }
I20260619 00:59:46.280593 2873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:46.281849 2879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:46.285977 2879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:46.294608 2879 catalog_manager.cc:1347] Generated new cluster ID: 0117e537fc534671a484308e5d058d7a
I20260619 00:59:46.294732 2879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:46.311009 2879 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:46.312215 2879 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:46.327739 2879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077e0e8bf32048e3b319a560a916230e: Generated new TSK 0
I20260619 00:59:46.328526 2879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:46.374962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:46.711691 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:46.711978 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:46.712071 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:46.719425 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:46.719576 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:46.719810 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:46.729475 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:46.730660 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:46.731948 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:46.745086 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:46.745034 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:46.745009 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:46.746043 2893 server_base.cc:1034] running on GCE node
I20260619 00:59:46.746652 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:46.747553 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:46.748723 2893 hybrid_clock.cc:648] HybridClock initialized: now 1781830786748662 us; error 53 us; skew 500 ppm
I20260619 00:59:46.748980 2893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:46 dist-test-slave-3rxd krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:46.752234 2893 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:46.753624 2893 webserver.cc:460] Webserver started at http://127.0.97.1:35473/ using document root <none> and password file <none>
I20260619 00:59:46.754269 2893 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:46.754395 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:46.754761 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:46.757570 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "7c10d91a98274cd2a34382c5d0bc9f1a"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "d02111a6e236e9955bce14ea26577461"
server_key_iv: "67c920190932c4b5a84e8bc0a7021820"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.758329 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7c10d91a98274cd2a34382c5d0bc9f1a"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "d02111a6e236e9955bce14ea26577461"
server_key_iv: "67c920190932c4b5a84e8bc0a7021820"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.763370 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 00:59:46.766672 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:46.768162 2893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:59:46.768321 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "7c10d91a98274cd2a34382c5d0bc9f1a"
format_stamp: "Formatted at 2026-06-19 00:59:46 on dist-test-slave-3rxd"
server_key: "d02111a6e236e9955bce14ea26577461"
server_key_iv: "67c920190932c4b5a84e8bc0a7021820"
server_key_version: "encryptionkey@0"
I20260619 00:59:46.768527 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:46.784040 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:46.788012 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:46.788239 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:46.789083 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:46.790455 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:46.790529 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:46.790688 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:46.790807 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 00:59:46.821815 2893 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42261
I20260619 00:59:46.821854 3020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42261 every 8 connection(s)
I20260619 00:59:46.823206 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:46.824693 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 2893
I20260619 00:59:46.824898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:46.825496 388 external_mini_cluster.cc:1428] Setting key fa0b3b8cc81cc3bf71e43ec00c7d5e4b
Jun 19 00:59:46 dist-test-slave-3rxd krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:59:46.845096 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:46.825793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53357 (local address 127.0.97.62:36873)
0619 00:59:46.826255 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:46.826261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:46.829793 (+ 3532us) server_negotiation.cc:407] Connection header received
0619 00:59:46.830174 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:46.830178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:46.830237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:46.830328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:46.832460 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.833136 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.834851 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.835260 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.835359 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:46.839526 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:46.839532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:46.839536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:46.839566 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:46.842081 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:46.842848 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:46.842858 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:46.842861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:46.842978 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:46.843401 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:46.843406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:46.843410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:46.843656 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:46.843818 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:46.844508 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 00:59:46.844724 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":231,"threads_started":1}
I20260619 00:59:46.846298 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:46.826415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36873 (local address 127.0.97.1:53357)
0619 00:59:46.827217 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:46.827287 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:46.829861 (+ 2574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:46.830598 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:46.830612 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:46.831442 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:46.832265 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:46.832292 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.833305 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.833312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:46.834662 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:46.834674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:46.835492 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:46.835501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:46.835834 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:46.836558 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:46.836593 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:46.839281 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:46.842231 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:46.842240 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:46.842256 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:46.842680 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:46.843089 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:46.843094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:46.843098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:46.843285 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:46.843780 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:46.843790 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:46.844131 (+ 341us) client_negotiation.cc:769] Sending connection context
0619 00:59:46.844392 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 00:59:46.844950 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:59:46.847995 3021 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36873
I20260619 00:59:46.848325 3021 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:46.849071 3021 heartbeater.cc:511] Master 127.0.97.62:36873 requested a full tablet report, sending...
I20260619 00:59:46.851308 2832 ts_manager.cc:194] Registered new tserver with Master: 7c10d91a98274cd2a34382c5d0bc9f1a (127.0.97.1:42261)
I20260619 00:59:46.852795 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:46.869637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:47.186208 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:47.186488 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:47.186580 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:47.193774 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:47.193893 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:47.194013 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:47.203706 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:47.204869 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:47.206133 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:47.218906 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:47.219002 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:47.218979 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:47.220065 3027 server_base.cc:1034] running on GCE node
I20260619 00:59:47.220625 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:47.221488 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:47.222770 3027 hybrid_clock.cc:648] HybridClock initialized: now 1781830787222687 us; error 51 us; skew 500 ppm
I20260619 00:59:47.223074 3027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:47 dist-test-slave-3rxd krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:47.226186 3027 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:47.227562 3027 webserver.cc:460] Webserver started at http://127.0.97.2:38937/ using document root <none> and password file <none>
I20260619 00:59:47.228201 3027 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:47.228296 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:47.228540 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:47.231426 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "f91f78187eef42e8849ce913cc51afa9"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "976a1f069af097d4baefbd107f535d66"
server_key_iv: "efcb0798a709c28697b8613b3cdb4cee"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.232138 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f91f78187eef42e8849ce913cc51afa9"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "976a1f069af097d4baefbd107f535d66"
server_key_iv: "efcb0798a709c28697b8613b3cdb4cee"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.237308 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 00:59:47.241452 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.242960 3027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 00:59:47.243139 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "f91f78187eef42e8849ce913cc51afa9"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "976a1f069af097d4baefbd107f535d66"
server_key_iv: "efcb0798a709c28697b8613b3cdb4cee"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.243362 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:47.269562 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:47.274011 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:47.274268 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:47.275218 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:47.276520 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:47.276588 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.276667 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:47.276710 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.308329 3027 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44233
I20260619 00:59:47.308418 3154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44233 every 8 connection(s)
I20260619 00:59:47.309631 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:47.317111 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3027
I20260619 00:59:47.317289 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:47.317690 388 external_mini_cluster.cc:1428] Setting key bd40352cb0dabdfe90c5973a5579774c
Jun 19 00:59:47 dist-test-slave-3rxd krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:47.330166 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:47.312095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33857 (local address 127.0.97.62:36873)
0619 00:59:47.312260 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:47.312265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:47.313344 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 00:59:47.315187 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:47.315192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:47.315284 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:47.315390 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:47.316827 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.317646 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.319360 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.319864 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.319982 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:47.323957 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:47.323962 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:47.323965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:47.323994 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:47.327295 (+ 3301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:47.327950 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:47.327957 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:47.327965 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:47.328094 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:47.328534 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:47.328539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:47.328542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:47.328780 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:47.328928 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:47.329585 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 00:59:47.329823 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:59:47.331180 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:47.312535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36873 (local address 127.0.97.2:33857)
0619 00:59:47.313196 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:47.313232 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:47.314922 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:47.315614 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:47.315623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:47.316102 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:47.316636 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:47.316653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.317784 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.317791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:47.319189 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:47.319200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.320088 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.320096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:47.320416 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:47.321155 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:47.321189 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:47.323749 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:47.327421 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:47.327428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:47.327440 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:47.327822 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:47.328227 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:47.328234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:47.328237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:47.328415 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:47.328901 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:47.328909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:47.329236 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 00:59:47.329485 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 00:59:47.330034 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":252,"threads_started":1}
I20260619 00:59:47.332659 3155 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36873
I20260619 00:59:47.332973 3155 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:47.333631 3155 heartbeater.cc:511] Master 127.0.97.62:36873 requested a full tablet report, sending...
I20260619 00:59:47.335088 2832 ts_manager.cc:194] Registered new tserver with Master: f91f78187eef42e8849ce913cc51afa9 (127.0.97.2:44233)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:59:47.335865 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:47.359799 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:47.670403 3160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:47.670662 3160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:47.670874 3160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:47.678048 3160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:47.678176 3160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:47.678354 3160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:47.687553 3160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:47.688730 3160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:47.690007 3160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:47.702878 3168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:47.702855 3166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:47.703384 3165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:47.703562 3160 server_base.cc:1034] running on GCE node
I20260619 00:59:47.704120 3160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:47.704943 3160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:47.706153 3160 hybrid_clock.cc:648] HybridClock initialized: now 1781830787706103 us; error 47 us; skew 500 ppm
I20260619 00:59:47.706437 3160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:47 dist-test-slave-3rxd krb5kdc[2787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:47.718066 3160 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:47.719477 3160 webserver.cc:460] Webserver started at http://127.0.97.3:40111/ using document root <none> and password file <none>
I20260619 00:59:47.720101 3160 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:47.720212 3160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:47.720470 3160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:47.723263 3160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "27caab1222cc413c81a9f95e8e6b6325"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "894cecc6617b69410e18ccb5f2e54953"
server_key_iv: "b3d44707c0a47c106296ff9736ac26c1"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.723985 3160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "27caab1222cc413c81a9f95e8e6b6325"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "894cecc6617b69410e18ccb5f2e54953"
server_key_iv: "b3d44707c0a47c106296ff9736ac26c1"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.728906 3160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:47.732216 3175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.733461 3160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:59:47.733623 3160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "27caab1222cc413c81a9f95e8e6b6325"
format_stamp: "Formatted at 2026-06-19 00:59:47 on dist-test-slave-3rxd"
server_key: "894cecc6617b69410e18ccb5f2e54953"
server_key_iv: "b3d44707c0a47c106296ff9736ac26c1"
server_key_version: "encryptionkey@0"
I20260619 00:59:47.733829 3160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:47.752926 3160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:47.756600 3160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:47.756875 3160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:47.757784 3160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:47.759114 3160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:47.759184 3160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.759351 3160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:47.759399 3160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:47.791168 3160 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35967
I20260619 00:59:47.791240 3287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35967 every 8 connection(s)
I20260619 00:59:47.792505 3160 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:47.795729 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3160
I20260619 00:59:47.795914 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:47.796481 388 external_mini_cluster.cc:1428] Setting key a366c6ec4b51436b2432e69fd8cf6379
Jun 19 00:59:47 dist-test-slave-3rxd krb5kdc[2787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:47.812343 3023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:47.795164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59297 (local address 127.0.97.62:36873)
0619 00:59:47.795384 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:47.795388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:47.796622 (+ 1234us) server_negotiation.cc:407] Connection header received
0619 00:59:47.798698 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:47.798703 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:47.798805 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:47.798921 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:47.800608 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.801342 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.802883 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.803309 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.803445 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:47.806804 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:47.806810 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:47.806814 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:47.806843 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:47.809189 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:47.809905 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:47.809909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:47.809911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:47.810075 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:47.810552 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:47.810557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:47.810560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:47.810895 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:47.811067 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:47.811799 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 00:59:47.812001 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:59:47.813207 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:47.795730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36873 (local address 127.0.97.3:59297)
0619 00:59:47.796459 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:47.796495 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:47.798438 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:47.799312 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:47.799326 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:47.799847 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:47.800434 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:47.800452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.801505 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.801517 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:47.802697 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:47.802708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:47.803561 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:47.803569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:47.803796 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:47.804478 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:47.804503 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:47.806567 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:47.809327 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:47.809335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:47.809350 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:47.809757 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:47.810206 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:47.810210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:47.810214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:47.810437 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:47.811038 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:47.811045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:47.811415 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 00:59:47.811705 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 00:59:47.812187 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":300,"threads_started":1}
I20260619 00:59:47.814481 3288 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36873
I20260619 00:59:47.814805 3288 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:47.815469 3288 heartbeater.cc:511] Master 127.0.97.62:36873 requested a full tablet report, sending...
I20260619 00:59:47.816736 2832 ts_manager.cc:194] Registered new tserver with Master: 27caab1222cc413c81a9f95e8e6b6325 (127.0.97.3:35967)
I20260619 00:59:47.817447 2832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59297
I20260619 00:59:47.823786 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:59:47.855823 3021 heartbeater.cc:503] Master 127.0.97.62:36873 was elected leader, sending a full tablet report...
W20260619 00:59:47.863133 3023 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 00:59:47.860637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44160 (local address 127.0.97.62:36873)
0619 00:59:47.860864 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:47.860872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:47.861320 (+ 448us) server_negotiation.cc:407] Connection header received
0619 00:59:47.861532 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:47.861538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:47.861613 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:47.861711 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:47.862941 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44160: BlockingRecv error: recv got EOF from 127.0.0.1:44160 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:59:47.868265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2893
W20260619 00:59:47.871294 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2893: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42261: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:59:47.871462 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2893
I20260619 00:59:47.890552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3027
W20260619 00:59:47.893111 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3027: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:44233: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:59:47.893275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3027
I20260619 00:59:47.907846 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3160
W20260619 00:59:47.910507 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3160: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:35967: server requires authentication, but client does not have Kerberos credentials available
I20260619 00:59:47.910645 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3160
I20260619 00:59:47.925524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 2800
I20260619 00:59:47.926651 2862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:48.026377 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 2800
2026-06-19T00:59:48Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3448 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:48 dist-test-slave-3rxd krb5kdc[3305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:48 dist-test-slave-3rxd krb5kdc[3305](info): set up 2 sockets
Jun 19 00:59:48 dist-test-slave-3rxd krb5kdc[3305](info): commencing operation
krb5kdc: starting...
W20260619 00:59:49.103188 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.008s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:49 dist-test-slave-3rxd krb5kdc[3305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:49.282341 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:49.605162 3318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:49.605456 3318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:49.605657 3318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:49.612874 3318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:49.612983 3318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:49.613104 3318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:49.613219 3318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:49.613353 3318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:49.623170 3318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:49.624473 3318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:49.625819 3318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:49.637897 3323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:49.637856 3324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:49.638742 3326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:49.638799 3318 server_base.cc:1034] running on GCE node
I20260619 00:59:49.639635 3318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:49.641098 3318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:49.642359 3318 hybrid_clock.cc:648] HybridClock initialized: now 1781830789642295 us; error 38 us; skew 500 ppm
I20260619 00:59:49.642839 3318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:49 dist-test-slave-3rxd krb5kdc[3305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:49.649964 3318 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:49.651551 3318 webserver.cc:460] Webserver started at http://127.0.97.62:33915/ using document root <none> and password file <none>
I20260619 00:59:49.652216 3318 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:49.652316 3318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:49.652578 3318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:49.655619 3318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "67b9fcef493140fd9fb960a89a93d8e2"
format_stamp: "Formatted at 2026-06-19 00:59:49 on dist-test-slave-3rxd"
server_key: "329d3347bde3f1da31f5838a44608fd4"
server_key_iv: "7a166d4dea0dc0e7206de66cd4abbe03"
server_key_version: "encryptionkey@0"
I20260619 00:59:49.656320 3318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "67b9fcef493140fd9fb960a89a93d8e2"
format_stamp: "Formatted at 2026-06-19 00:59:49 on dist-test-slave-3rxd"
server_key: "329d3347bde3f1da31f5838a44608fd4"
server_key_iv: "7a166d4dea0dc0e7206de66cd4abbe03"
server_key_version: "encryptionkey@0"
I20260619 00:59:49.661267 3318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:49.664507 3333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:49.665885 3318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:59:49.666071 3318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "67b9fcef493140fd9fb960a89a93d8e2"
format_stamp: "Formatted at 2026-06-19 00:59:49 on dist-test-slave-3rxd"
server_key: "329d3347bde3f1da31f5838a44608fd4"
server_key_iv: "7a166d4dea0dc0e7206de66cd4abbe03"
server_key_version: "encryptionkey@0"
I20260619 00:59:49.666263 3318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:49.685086 3318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:49.688769 3318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:49.689047 3318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:49.706061 3318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46835
I20260619 00:59:49.706099 3384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46835 every 8 connection(s)
I20260619 00:59:49.707389 3318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:49.709044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3318
I20260619 00:59:49.709233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:49.709878 388 external_mini_cluster.cc:1428] Setting key 18b7196d97c9dbf01bdfa9a06e4aa5fe
I20260619 00:59:49.711696 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 00:59:49 dist-test-slave-3rxd krb5kdc[3305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:49.724771 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2: Bootstrap starting.
I20260619 00:59:49.728842 3385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:49.730181 3385 log.cc:824] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:49.732946 3385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2: No bootstrap required, opened a new log
I20260619 00:59:49.732990 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:49.711721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.0.97.62:46835)
0619 00:59:49.712536 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:49.712551 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:49.712601 (+ 50us) server_negotiation.cc:407] Connection header received
0619 00:59:49.714837 (+ 2236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:49.714862 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:49.715339 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:49.715777 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:49.717014 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:49.719061 (+ 2047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:49.720223 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:49.721214 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:49.721408 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:49.725254 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:49.725264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:49.725281 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:49.725324 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:49.729120 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:49.729777 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:49.729786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:49.729795 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:49.730023 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:49.730370 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:49.730380 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:49.730382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:49.730973 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:49.731274 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:49.731693 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 00:59:49.732058 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":331,"threads_started":1}
I20260619 00:59:49.736541 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 00:59:49.740326 3385 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260619 00:59:49.740751 3385 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:49.740850 3385 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67b9fcef493140fd9fb960a89a93d8e2, State: Initialized, Role: FOLLOWER
I20260619 00:59:49.741446 3385 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260619 00:59:49.741677 3385 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:49.741820 3385 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:49.742051 3385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:49.745348 3385 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260619 00:59:49.745908 3385 leader_election.cc:302] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67b9fcef493140fd9fb960a89a93d8e2; no voters:
I20260619 00:59:49.746526 3385 leader_election.cc:288] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:49.746649 3391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:49.748072 3391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [term 1 LEADER]: Becoming Leader. State: Replica: 67b9fcef493140fd9fb960a89a93d8e2, State: Running, Role: LEADER
I20260619 00:59:49.748708 3385 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:49.748587 3391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } }
I20260619 00:59:49.751346 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67b9fcef493140fd9fb960a89a93d8e2. Latest consensus state: current_term: 1 leader_uuid: "67b9fcef493140fd9fb960a89a93d8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } } }
I20260619 00:59:49.751744 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:49.751744 3392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67b9fcef493140fd9fb960a89a93d8e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67b9fcef493140fd9fb960a89a93d8e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46835 } } }
I20260619 00:59:49.752032 3392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:49.760567 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:49.766048 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:49.778486 3411 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 00:59:49.780535 3398 catalog_manager.cc:1347] Generated new cluster ID: ad91163eed664b38949cf72ba84a9c22
I20260619 00:59:49.780653 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:49.790637 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:49.781475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.97.62:46835)
0619 00:59:49.781672 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:49.781679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:49.782278 (+ 599us) server_negotiation.cc:407] Connection header received
0619 00:59:49.782457 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:49.782464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:49.782579 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:49.782761 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:49.783874 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:49.785039 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:49.786788 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:49.787459 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:49.787606 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:49.788740 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:49.788747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:49.788754 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 00:59:49.789606 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:49.789687 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 00:59:49.789722 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:49.789829 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:49.790000 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 00:59:49.790098 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 00:59:49.811152 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:49.813048 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:49.825212 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67b9fcef493140fd9fb960a89a93d8e2: Generated new TSK 0
I20260619 00:59:49.826243 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 00:59:49.837863 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3318
I20260619 00:59:49.838977 3380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:49.928467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3318
2026-06-19T00:59:49Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1897 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:50 dist-test-slave-3rxd krb5kdc[3419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:50 dist-test-slave-3rxd krb5kdc[3419](info): set up 2 sockets
Jun 19 00:59:50 dist-test-slave-3rxd krb5kdc[3419](info): commencing operation
krb5kdc: starting...
W20260619 00:59:51.003065 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:51 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:51.185983 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42089
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:51.498137 3432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:51.498476 3432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:51.498657 3432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:51.505867 3432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:51.506011 3432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:51.506116 3432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:51.506206 3432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:51.506270 3432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:51.521518 3432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:51.523010 3432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:51.524305 3432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:51.538518 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:51.538757 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:51.538725 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:51.539481 3432 server_base.cc:1034] running on GCE node
I20260619 00:59:51.540467 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:51.541915 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:51.543155 3432 hybrid_clock.cc:648] HybridClock initialized: now 1781830791543110 us; error 43 us; skew 500 ppm
I20260619 00:59:51.543498 3432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:51 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:51.546643 3432 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:51.548096 3432 webserver.cc:460] Webserver started at http://127.0.97.62:39399/ using document root <none> and password file <none>
I20260619 00:59:51.548785 3432 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:51.548928 3432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:51.549232 3432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:51.552277 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "e2a94432fee74eefa19e54d01ad1f193"
format_stamp: "Formatted at 2026-06-19 00:59:51 on dist-test-slave-3rxd"
server_key: "d4526a648c9d360d161c3cab211341e4"
server_key_iv: "c1a4184fd76fcc32fcfff61e91342d7c"
server_key_version: "encryptionkey@0"
I20260619 00:59:51.553030 3432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "e2a94432fee74eefa19e54d01ad1f193"
format_stamp: "Formatted at 2026-06-19 00:59:51 on dist-test-slave-3rxd"
server_key: "d4526a648c9d360d161c3cab211341e4"
server_key_iv: "c1a4184fd76fcc32fcfff61e91342d7c"
server_key_version: "encryptionkey@0"
I20260619 00:59:51.557989 3432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 00:59:51.561295 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:51.562691 3432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:59:51.562908 3432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "e2a94432fee74eefa19e54d01ad1f193"
format_stamp: "Formatted at 2026-06-19 00:59:51 on dist-test-slave-3rxd"
server_key: "d4526a648c9d360d161c3cab211341e4"
server_key_iv: "c1a4184fd76fcc32fcfff61e91342d7c"
server_key_version: "encryptionkey@0"
I20260619 00:59:51.563102 3432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:51.576478 3432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:51.580262 3432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:51.580549 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:51.598127 3432 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42089
I20260619 00:59:51.598155 3498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42089 every 8 connection(s)
I20260619 00:59:51.599509 3432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:51.602636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3432
I20260619 00:59:51.602872 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:51.603477 3499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:51.603502 388 external_mini_cluster.cc:1428] Setting key fe78404ea6b71c273c3616810b396bce
I20260619 00:59:51.617260 3499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193: Bootstrap starting.
Jun 19 00:59:51 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:51.620940 3499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:51.621969 3499 log.cc:824] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:51.624696 3499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193: No bootstrap required, opened a new log
I20260619 00:59:51.625887 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:51.605417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.0.97.62:42089)
0619 00:59:51.606213 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:51.606227 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:51.606280 (+ 53us) server_negotiation.cc:407] Connection header received
0619 00:59:51.608225 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:51.608252 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:51.608779 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:51.609199 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:51.610519 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:51.612436 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:51.613650 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:51.614739 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:51.614921 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:51.619064 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:51.619071 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:51.619088 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:51.619129 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:51.622091 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:51.622768 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:51.622776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:51.622784 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:51.622949 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:51.623333 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:51.623340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:51.623343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:51.623808 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:51.624140 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:51.624534 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 00:59:51.624946 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":314,"threads_started":1}
I20260619 00:59:51.632730 3499 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } }
I20260619 00:59:51.633090 3499 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:51.633191 3499 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a94432fee74eefa19e54d01ad1f193, State: Initialized, Role: FOLLOWER
I20260619 00:59:51.633699 3499 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } }
I20260619 00:59:51.633929 3499 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:51.634073 3499 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:51.634234 3499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:51.636513 3499 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } }
I20260619 00:59:51.636987 3499 leader_election.cc:302] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a94432fee74eefa19e54d01ad1f193; no voters:
I20260619 00:59:51.637526 3499 leader_election.cc:288] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:51.637660 3504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:51.638319 3504 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [term 1 LEADER]: Becoming Leader. State: Replica: e2a94432fee74eefa19e54d01ad1f193, State: Running, Role: LEADER
I20260619 00:59:51.638882 3504 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } }
I20260619 00:59:51.639605 3499 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:51.641970 3506 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a94432fee74eefa19e54d01ad1f193. Latest consensus state: current_term: 1 leader_uuid: "e2a94432fee74eefa19e54d01ad1f193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } } }
I20260619 00:59:51.641937 3505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a94432fee74eefa19e54d01ad1f193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a94432fee74eefa19e54d01ad1f193" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42089 } } }
I20260619 00:59:51.642309 3506 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:51.642309 3505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:51.643865 3511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:51.650126 3511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:51.659013 3511 catalog_manager.cc:1347] Generated new cluster ID: 335c84950c5d424f8e689626b87e2f71
I20260619 00:59:51.659121 3511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:51.687278 3511 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:51.688714 3511 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:51.699705 3511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a94432fee74eefa19e54d01ad1f193: Generated new TSK 0
I20260619 00:59:51.700591 3511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:51.757931 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:52.098073 3525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:52.098336 3525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:52.098428 3525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:52.105530 3525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:52.105628 3525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:52.105744 3525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:52.115096 3525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:52.116322 3525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:52.117542 3525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:52.130128 3530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:52.130079 3531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:52.130154 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:52.131397 3525 server_base.cc:1034] running on GCE node
I20260619 00:59:52.131980 3525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:52.132844 3525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:52.134106 3525 hybrid_clock.cc:648] HybridClock initialized: now 1781830792134064 us; error 31 us; skew 500 ppm
I20260619 00:59:52.134414 3525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:52 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:52.137617 3525 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:52.139057 3525 webserver.cc:460] Webserver started at http://127.0.97.1:34099/ using document root <none> and password file <none>
I20260619 00:59:52.139745 3525 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:52.139889 3525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:52.140138 3525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:52.143024 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b2f5bc6632d4619a311ad37d682e927"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "4c27543d0adc438dc5f96aaa82805490"
server_key_iv: "b0e8b1192b2719d9e07187321c2f5896"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.143741 3525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b2f5bc6632d4619a311ad37d682e927"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "4c27543d0adc438dc5f96aaa82805490"
server_key_iv: "b0e8b1192b2719d9e07187321c2f5896"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.149142 3525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:52.152488 3540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.153725 3525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 00:59:52.153895 3525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "3b2f5bc6632d4619a311ad37d682e927"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "4c27543d0adc438dc5f96aaa82805490"
server_key_iv: "b0e8b1192b2719d9e07187321c2f5896"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.154093 3525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:52.174713 3525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:52.178435 3525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:52.178766 3525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:52.179679 3525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:52.180915 3525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:52.181017 3525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.181167 3525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:52.181254 3525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.213433 3525 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34439
I20260619 00:59:52.213495 3652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34439 every 8 connection(s)
I20260619 00:59:52.214648 3525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:52.217927 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3525
I20260619 00:59:52.218169 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:52.218884 388 external_mini_cluster.cc:1428] Setting key 660d7e1720f669a7efd34080a8aa7eba
Jun 19 00:59:52 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:52.234453 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:52.217063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44071 (local address 127.0.97.62:42089)
0619 00:59:52.217456 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:52.217461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:52.218562 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 00:59:52.220506 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:52.220511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:52.220592 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:52.220701 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:52.222061 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.222835 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.224323 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.224713 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.224845 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:52.228762 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:52.228767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:52.228770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:52.228800 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:52.231317 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:52.232220 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:52.232224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:52.232226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:52.232330 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:52.232788 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:52.232791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:52.232793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:52.233035 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:52.233145 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:52.234060 (+ 915us) server_negotiation.cc:299] Negotiation successful
0619 00:59:52.234224 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":205,"threads_started":1}
I20260619 00:59:52.236058 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:52.217616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42089 (local address 127.0.97.1:44071)
0619 00:59:52.218386 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:52.218436 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:52.220263 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:52.220919 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:52.220927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:52.221378 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:52.221895 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:52.221911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.222975 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.222983 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:52.224174 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:52.224185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.224928 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.224937 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:52.225165 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:52.225910 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:52.225935 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:52.228561 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:52.231448 (+ 2887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:52.231459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:52.231478 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:52.232105 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:52.232436 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:52.232442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:52.232447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:52.232683 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:52.233161 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:52.233172 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:52.233647 (+ 475us) client_negotiation.cc:769] Sending connection context
0619 00:59:52.233987 (+ 340us) client_negotiation.cc:240] Negotiation successful
0619 00:59:52.234547 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:59:52.237613 3653 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42089
I20260619 00:59:52.237910 3653 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:52.238623 3653 heartbeater.cc:511] Master 127.0.97.62:42089 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:59:52.240815 3464 ts_manager.cc:194] Registered new tserver with Master: 3b2f5bc6632d4619a311ad37d682e927 (127.0.97.1:34439)
I20260619 00:59:52.242233 3464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:52.262275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:52.571853 3659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:52.572149 3659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:52.572319 3659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:52.579242 3659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:52.579381 3659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:52.579555 3659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:52.588853 3659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:52.590066 3659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:52.591367 3659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:52.604066 3664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:52.604036 3665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:52.604261 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:52.604925 3659 server_base.cc:1034] running on GCE node
I20260619 00:59:52.605523 3659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:52.606462 3659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:52.607684 3659 hybrid_clock.cc:648] HybridClock initialized: now 1781830792607613 us; error 67 us; skew 500 ppm
I20260619 00:59:52.608018 3659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:52 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:52.611066 3659 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:52.612468 3659 webserver.cc:460] Webserver started at http://127.0.97.2:44787/ using document root <none> and password file <none>
I20260619 00:59:52.613107 3659 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:52.613232 3659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:52.613528 3659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:52.616255 3659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "25ff3dc208d04ec08181f9007dd1c6a7"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "98b8c99624e9b58392394b5df298a652"
server_key_iv: "7f92f49a1290d09f8512273bd3342036"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.617026 3659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "25ff3dc208d04ec08181f9007dd1c6a7"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "98b8c99624e9b58392394b5df298a652"
server_key_iv: "7f92f49a1290d09f8512273bd3342036"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.621722 3659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 00:59:52.625084 3674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.626251 3659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 00:59:52.626402 3659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "25ff3dc208d04ec08181f9007dd1c6a7"
format_stamp: "Formatted at 2026-06-19 00:59:52 on dist-test-slave-3rxd"
server_key: "98b8c99624e9b58392394b5df298a652"
server_key_iv: "7f92f49a1290d09f8512273bd3342036"
server_key_version: "encryptionkey@0"
I20260619 00:59:52.626593 3659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:52.647046 3659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:52.651264 3659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:52.651558 3659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:52.652446 3659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:52.653720 3659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:52.653793 3659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.653960 3659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:52.654013 3659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:52.686460 3659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43781
I20260619 00:59:52.686509 3786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43781 every 8 connection(s)
I20260619 00:59:52.687785 3659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:52.689273 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3659
I20260619 00:59:52.689463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:52.689977 388 external_mini_cluster.cc:1428] Setting key b292e3bc0ec39fa9b8136177d8b28c78
Jun 19 00:59:52 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:59:52.709439 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:52.690466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43489 (local address 127.0.97.62:42089)
0619 00:59:52.690676 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:52.690682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:52.692129 (+ 1447us) server_negotiation.cc:407] Connection header received
0619 00:59:52.695026 (+ 2897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:52.695030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:52.695104 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:52.695183 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:52.697248 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.697956 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.699613 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.700107 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.700243 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:52.703865 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:52.703870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:52.703873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:52.703903 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:52.706353 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:52.707275 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:52.707279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:52.707281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:52.707400 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:52.707854 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:52.707857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:52.707859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:52.708097 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:52.708208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:52.709057 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 00:59:52.709211 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 00:59:52.710469 3789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:52.691117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42089 (local address 127.0.97.2:43489)
0619 00:59:52.691952 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:52.692014 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:52.694674 (+ 2660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:52.695482 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:52.695497 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:52.696265 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:52.697072 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:52.697104 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.698122 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.698129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:52.699461 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:52.699472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:52.700325 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:52.700332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:52.700573 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:52.701250 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:52.701271 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:52.703640 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:52.706489 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:52.706500 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:52.706519 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:52.707155 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:52.707513 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:52.707518 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:52.707521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:52.707747 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:52.708229 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:52.708238 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:52.708646 (+ 408us) client_negotiation.cc:769] Sending connection context
0619 00:59:52.708958 (+ 312us) client_negotiation.cc:240] Negotiation successful
0619 00:59:52.709370 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 00:59:52.712007 3787 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42089
I20260619 00:59:52.712268 3787 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:52.712904 3787 heartbeater.cc:511] Master 127.0.97.62:42089 requested a full tablet report, sending...
I20260619 00:59:52.714236 3464 ts_manager.cc:194] Registered new tserver with Master: 25ff3dc208d04ec08181f9007dd1c6a7 (127.0.97.2:43781)
I20260619 00:59:52.715099 3464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:52.734474 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:53.050900 3792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:53.051162 3792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:53.051254 3792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:53.058295 3792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:53.058390 3792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:53.058502 3792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:53.067869 3792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:53.069067 3792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:53.070302 3792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:53.082855 3798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:53.082895 3800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:53.082831 3797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:53.083590 3792 server_base.cc:1034] running on GCE node
I20260619 00:59:53.084115 3792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:53.084949 3792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:53.086172 3792 hybrid_clock.cc:648] HybridClock initialized: now 1781830793086119 us; error 50 us; skew 500 ppm
I20260619 00:59:53.086488 3792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:53.089433 3792 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:53.090992 3792 webserver.cc:460] Webserver started at http://127.0.97.3:41883/ using document root <none> and password file <none>
I20260619 00:59:53.091650 3792 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:53.091743 3792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:53.091984 3792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:53.094647 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "3f49407dbef1461dac9b4ebf7357930a"
format_stamp: "Formatted at 2026-06-19 00:59:53 on dist-test-slave-3rxd"
server_key: "5dc51da30c8362e72705ca1355ae4994"
server_key_iv: "43206b0ed4af08eeb4c3f18afde746e0"
server_key_version: "encryptionkey@0"
I20260619 00:59:53.095415 3792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3f49407dbef1461dac9b4ebf7357930a"
format_stamp: "Formatted at 2026-06-19 00:59:53 on dist-test-slave-3rxd"
server_key: "5dc51da30c8362e72705ca1355ae4994"
server_key_iv: "43206b0ed4af08eeb4c3f18afde746e0"
server_key_version: "encryptionkey@0"
I20260619 00:59:53.100049 3792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 00:59:53.103363 3807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:53.104679 3792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 00:59:53.104804 3792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "3f49407dbef1461dac9b4ebf7357930a"
format_stamp: "Formatted at 2026-06-19 00:59:53 on dist-test-slave-3rxd"
server_key: "5dc51da30c8362e72705ca1355ae4994"
server_key_iv: "43206b0ed4af08eeb4c3f18afde746e0"
server_key_version: "encryptionkey@0"
I20260619 00:59:53.104948 3792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:53.118059 3792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:53.121747 3792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:53.121974 3792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:53.122877 3792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:53.124156 3792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:53.124228 3792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:53.124393 3792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:53.124467 3792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:53.156363 3792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45483
I20260619 00:59:53.156397 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45483 every 8 connection(s)
I20260619 00:59:53.157621 3792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:53.161769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3792
I20260619 00:59:53.161938 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:53.162546 388 external_mini_cluster.cc:1428] Setting key 77ef378926a948cd0d2fe0397f8463be
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:53.176966 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.160107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34547 (local address 127.0.97.62:42089)
0619 00:59:53.160313 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:53.160320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:53.161385 (+ 1065us) server_negotiation.cc:407] Connection header received
0619 00:59:53.163772 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:53.163777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:53.163865 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:53.163971 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:53.165348 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.166135 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.167703 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.168105 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.168237 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.171477 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:53.171482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:53.171485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:53.171517 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:53.174068 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.174775 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.174784 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.174787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.174906 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.175332 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.175337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.175340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.175601 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:53.175753 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:53.176434 (+ 681us) server_negotiation.cc:299] Negotiation successful
0619 00:59:53.176636 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 00:59:53.177932 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.160534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42089 (local address 127.0.97.3:34547)
0619 00:59:53.161210 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:53.161250 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:53.163509 (+ 2259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:53.164194 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:53.164202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:53.164672 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:53.165197 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:53.165212 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.166266 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.166274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:53.167567 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:53.167577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.168313 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.168322 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:53.168563 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.169210 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:53.169232 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:53.171270 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:53.174211 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:53.174220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:53.174236 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:53.174634 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:53.175015 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:53.175026 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:53.175032 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:53.175215 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:53.175727 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:53.175735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:53.176049 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 00:59:53.176292 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 00:59:53.176799 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":263,"threads_started":1}
I20260619 00:59:53.179193 3920 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42089
I20260619 00:59:53.179476 3920 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:53.180154 3920 heartbeater.cc:511] Master 127.0.97.62:42089 requested a full tablet report, sending...
I20260619 00:59:53.181466 3464 ts_manager.cc:194] Registered new tserver with Master: 3f49407dbef1461dac9b4ebf7357930a (127.0.97.3:45483)
I20260619 00:59:53.182202 3464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34547
I20260619 00:59:53.189867 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:53.234611 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.221950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.97.62:42089)
0619 00:59:53.222118 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:53.222123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:53.222643 (+ 520us) server_negotiation.cc:407] Connection header received
0619 00:59:53.222848 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:53.222852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:53.222912 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:53.223013 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:53.224242 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.224921 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.226064 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.226432 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.226516 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.230145 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:53.230150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:53.230153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:53.230218 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:53.232750 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.233259 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.233262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.233265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.233365 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.233652 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.233655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.233657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.233914 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:53.234094 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:53.234266 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 00:59:53.234400 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 00:59:53.235152 3464 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53464
I20260619 00:59:53.236780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3525
I20260619 00:59:53.244958 3653 heartbeater.cc:503] Master 127.0.97.62:42089 was elected leader, sending a full tablet report...
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:53.252230 3931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.237723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.97.1:34439)
0619 00:59:53.238300 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:53.238310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:53.238337 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:59:53.238392 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:53.238397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:53.238595 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:53.238776 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:53.239890 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.240953 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.242124 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.242664 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.242836 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.246988 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:53.246999 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:53.247013 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:53.247040 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:53.249778 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.250396 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.250403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.250405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.250535 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.250903 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.250906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.250908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.251230 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:53.251468 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:53.251646 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 00:59:53.251860 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":248,"threads_started":1}
W20260619 00:59:53.253352 3648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51370
W20260619 00:59:53.254415 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3525: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:53.254624 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3525
I20260619 00:59:53.270370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3659
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:53.284651 3932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.271183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36292 (local address 127.0.97.2:43781)
0619 00:59:53.271702 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:53.271709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:53.271730 (+ 21us) server_negotiation.cc:407] Connection header received
0619 00:59:53.271788 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:53.271794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:53.272003 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:53.272123 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:53.273171 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.274155 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.275382 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.275913 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.276007 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.279872 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:53.279882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:53.279895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:53.279931 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:53.282407 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.283048 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.283055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.283057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.283171 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.283456 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.283459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.283460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.283757 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:53.283959 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:53.284189 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 00:59:53.284368 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
W20260619 00:59:53.285885 3782 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36292
W20260619 00:59:53.286947 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:53.287096 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3659
I20260619 00:59:53.302209 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3792
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:53.318274 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:53.303123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.97.3:45483)
0619 00:59:53.303708 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:53.303714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:53.303741 (+ 27us) server_negotiation.cc:407] Connection header received
0619 00:59:53.303795 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:53.303801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:53.304002 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:53.304121 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:53.305288 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.306995 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.308225 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:53.308784 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:53.308879 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:53.313172 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:53.313182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:53.313196 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:53.313223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:53.315907 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.316484 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.316492 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.316494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.316602 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:53.316925 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:53.316928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:53.316930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:53.317219 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:53.317435 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:53.317687 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 00:59:53.317869 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":303,"threads_started":1}
W20260619 00:59:53.319690 3915 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41676
W20260619 00:59:53.320793 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3792: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 00:59:53.320952 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3792
I20260619 00:59:53.336035 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3432
I20260619 00:59:53.337126 3494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:53.437019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3432
2026-06-19T00:59:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3514 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3937](info): set up 2 sockets
Jun 19 00:59:53 dist-test-slave-3rxd krb5kdc[3937](info): commencing operation
krb5kdc: starting...
W20260619 00:59:54.523240 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:54 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:54.714413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33921
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:55.026082 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:55.026373 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:55.026538 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:55.035349 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:55.035480 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:55.035555 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:55.035640 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:55.035701 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:55.045614 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:55.047082 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:55.048481 3950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:55.060472 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:55.060526 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:55.060526 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:55.061460 3950 server_base.cc:1034] running on GCE node
I20260619 00:59:55.062256 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:55.063699 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:55.064906 3950 hybrid_clock.cc:648] HybridClock initialized: now 1781830795064853 us; error 39 us; skew 500 ppm
I20260619 00:59:55.065272 3950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:55 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:55.071456 3950 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:55.072984 3950 webserver.cc:460] Webserver started at http://127.0.97.62:34689/ using document root <none> and password file <none>
I20260619 00:59:55.073688 3950 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:55.073815 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:55.074150 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:55.077085 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "4dcc4f22d34049c3adf99327c3662c0d"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "4c0f462f18c4fbb724bc4c30a3177126"
server_key_iv: "82e74f94b3499259553d8f8c72e40b36"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.077850 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "4dcc4f22d34049c3adf99327c3662c0d"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "4c0f462f18c4fbb724bc4c30a3177126"
server_key_iv: "82e74f94b3499259553d8f8c72e40b36"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.082831 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 00:59:55.086067 3965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:55.087473 3950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 00:59:55.087659 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "4dcc4f22d34049c3adf99327c3662c0d"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "4c0f462f18c4fbb724bc4c30a3177126"
server_key_iv: "82e74f94b3499259553d8f8c72e40b36"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.087858 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:55.121747 3950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:55.125716 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:55.126004 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:55.143954 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33921
I20260619 00:59:55.143999 4016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33921 every 8 connection(s)
I20260619 00:59:55.145291 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:55.148890 4017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:55.152446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 3950
I20260619 00:59:55.152670 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:55.153421 388 external_mini_cluster.cc:1428] Setting key 66256c0532eed19d0e96661a893d5b0c
I20260619 00:59:55.161255 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d: Bootstrap starting.
I20260619 00:59:55.164358 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:55.165367 4017 log.cc:824] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d: Log is configured to *not* fsync() on all Append() calls
Jun 19 00:59:55 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:55.168475 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d: No bootstrap required, opened a new log
I20260619 00:59:55.176570 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:55.155539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.0.97.62:33921)
0619 00:59:55.156318 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:55.156348 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:55.156390 (+ 42us) server_negotiation.cc:407] Connection header received
0619 00:59:55.158004 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:55.158024 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:55.158411 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:55.158854 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:55.160283 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.161604 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.162909 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.163707 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.163927 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:55.168110 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:55.168121 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:55.168141 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:55.168188 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:55.172108 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:55.172800 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:55.172808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:55.172816 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:55.173028 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:55.173349 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:55.173353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:55.173356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:55.174075 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:55.174386 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:55.174670 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 00:59:55.175157 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":373,"threads_started":1}
I20260619 00:59:55.179209 4017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } }
I20260619 00:59:55.179653 4017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:55.179746 4017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dcc4f22d34049c3adf99327c3662c0d, State: Initialized, Role: FOLLOWER
I20260619 00:59:55.180318 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } }
I20260619 00:59:55.180553 4017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:55.180706 4017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:55.180864 4017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:55.183262 4017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } }
I20260619 00:59:55.183761 4017 leader_election.cc:302] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dcc4f22d34049c3adf99327c3662c0d; no voters:
I20260619 00:59:55.184294 4017 leader_election.cc:288] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:55.184428 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:55.185159 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [term 1 LEADER]: Becoming Leader. State: Replica: 4dcc4f22d34049c3adf99327c3662c0d, State: Running, Role: LEADER
I20260619 00:59:55.185632 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } }
I20260619 00:59:55.186318 4017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:55.188606 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dcc4f22d34049c3adf99327c3662c0d. Latest consensus state: current_term: 1 leader_uuid: "4dcc4f22d34049c3adf99327c3662c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } } }
I20260619 00:59:55.188956 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:55.189359 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dcc4f22d34049c3adf99327c3662c0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dcc4f22d34049c3adf99327c3662c0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33921 } } }
I20260619 00:59:55.189646 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:55.197423 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:55.203023 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:55.211094 4028 catalog_manager.cc:1347] Generated new cluster ID: 78fe94cdba7444328f0e809511ae54ba
I20260619 00:59:55.211213 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:55.224272 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:55.225500 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:55.234567 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dcc4f22d34049c3adf99327c3662c0d: Generated new TSK 0
I20260619 00:59:55.235399 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:55.286607 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:55.607414 4043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:55.607697 4043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:55.607888 4043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:55.615185 4043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:55.615303 4043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:55.615500 4043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:55.625274 4043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:55.626613 4043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:55.627987 4043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:55.641521 4048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:55.641464 4051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:55.641484 4049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:55.642354 4043 server_base.cc:1034] running on GCE node
I20260619 00:59:55.643105 4043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:55.644114 4043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:55.645342 4043 hybrid_clock.cc:648] HybridClock initialized: now 1781830795645311 us; error 61 us; skew 500 ppm
I20260619 00:59:55.645671 4043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:55 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:55.648838 4043 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:55.650326 4043 webserver.cc:460] Webserver started at http://127.0.97.1:43085/ using document root <none> and password file <none>
I20260619 00:59:55.651106 4043 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:55.651252 4043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:55.651568 4043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:55.654556 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "b4e179d8bae646aa983f177375cf7f43"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "b5e41c00060c4589bf70bdb73a2be31a"
server_key_iv: "d86f5d9a25a6290c1fe400612746f23c"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.655489 4043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b4e179d8bae646aa983f177375cf7f43"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "b5e41c00060c4589bf70bdb73a2be31a"
server_key_iv: "d86f5d9a25a6290c1fe400612746f23c"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.660441 4043 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:55.663786 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:55.665184 4043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:55.665374 4043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "b4e179d8bae646aa983f177375cf7f43"
format_stamp: "Formatted at 2026-06-19 00:59:55 on dist-test-slave-3rxd"
server_key: "b5e41c00060c4589bf70bdb73a2be31a"
server_key_iv: "d86f5d9a25a6290c1fe400612746f23c"
server_key_version: "encryptionkey@0"
I20260619 00:59:55.665592 4043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:55.679240 4043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:55.683573 4043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:55.683859 4043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:55.684762 4043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:55.686074 4043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:55.686170 4043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:55.686321 4043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:55.686396 4043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:55.718356 4043 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43371
I20260619 00:59:55.718397 4170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43371 every 8 connection(s)
I20260619 00:59:55.719672 4043 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:55.726305 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4043
I20260619 00:59:55.726518 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:55.727191 388 external_mini_cluster.cc:1428] Setting key 9fce362a2c266fa3955a979d1001c930
Jun 19 00:59:55 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:55.741015 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:55.722272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50385 (local address 127.0.97.62:33921)
0619 00:59:55.722789 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:55.722796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:55.723681 (+ 885us) server_negotiation.cc:407] Connection header received
0619 00:59:55.725431 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:55.725437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:55.725544 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:55.725663 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:55.727359 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.728282 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.730555 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.731012 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.731110 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:55.735356 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:55.735360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:55.735364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:55.735394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:55.737988 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:55.738740 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:55.738748 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:55.738754 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:55.738882 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:55.739310 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:55.739315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:55.739319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:55.739584 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:55.739743 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:55.740420 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 00:59:55.740642 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":265,"threads_started":1}
I20260619 00:59:55.742007 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:55.722813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33921 (local address 127.0.97.1:50385)
0619 00:59:55.723523 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:55.723560 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:55.725163 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:55.725917 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:55.725926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:55.726482 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:55.727186 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:55.727204 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.728433 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.728438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:55.730401 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:55.730419 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:55.731232 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:55.731242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:55.731581 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:55.732315 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:55.732350 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:55.735131 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:55.738142 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:55.738152 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:55.738169 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:55.738585 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:55.738991 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:55.738998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:55.739004 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:55.739190 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:55.739716 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:55.739724 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:55.740077 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 00:59:55.740335 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 00:59:55.740872 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":278,"threads_started":1}
I20260619 00:59:55.743672 4171 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33921
I20260619 00:59:55.743989 4171 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:55.744696 4171 heartbeater.cc:511] Master 127.0.97.62:33921 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:59:55.746830 3982 ts_manager.cc:194] Registered new tserver with Master: b4e179d8bae646aa983f177375cf7f43 (127.0.97.1:43371)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 00:59:55.748576 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:55.769985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:56.078951 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:56.079218 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:56.079367 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:56.086167 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:56.086282 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:56.086505 4177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:56.096217 4177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:56.097441 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:56.098672 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:56.111358 4183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:56.111330 4185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:56.111287 4182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:56.112152 4177 server_base.cc:1034] running on GCE node
I20260619 00:59:56.112809 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:56.113684 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:56.114902 4177 hybrid_clock.cc:648] HybridClock initialized: now 1781830796114853 us; error 62 us; skew 500 ppm
I20260619 00:59:56.115206 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:56.118239 4177 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:56.119663 4177 webserver.cc:460] Webserver started at http://127.0.97.2:37635/ using document root <none> and password file <none>
I20260619 00:59:56.120335 4177 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:56.120476 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:56.120771 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:56.123726 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "7a65e320b46043a6947a508ebce4cfdd"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "1ef306d5ec1be9bfb179fb6e1ca21013"
server_key_iv: "9dffd6ce400696c228b4cd032ad39fc1"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.124486 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7a65e320b46043a6947a508ebce4cfdd"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "1ef306d5ec1be9bfb179fb6e1ca21013"
server_key_iv: "9dffd6ce400696c228b4cd032ad39fc1"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.129359 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 00:59:56.132666 4192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:56.133965 4177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 00:59:56.134135 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "7a65e320b46043a6947a508ebce4cfdd"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "1ef306d5ec1be9bfb179fb6e1ca21013"
server_key_iv: "9dffd6ce400696c228b4cd032ad39fc1"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.134339 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:56.175683 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:56.180121 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:56.180418 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:56.181353 4177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:56.182669 4177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:56.182792 4177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 00:59:56.182943 4177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:56.183017 4177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:56.215091 4177 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36009
I20260619 00:59:56.215204 4304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36009 every 8 connection(s)
I20260619 00:59:56.216300 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:56.225901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4177
I20260619 00:59:56.226099 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:56.226578 388 external_mini_cluster.cc:1428] Setting key 34d92cffc631c3959b53d14436883a39
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:56.236582 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.218636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40813 (local address 127.0.97.62:33921)
0619 00:59:56.218819 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.218824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.219932 (+ 1108us) server_negotiation.cc:407] Connection header received
0619 00:59:56.221659 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.221664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.221750 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.221857 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.223281 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.223965 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.225458 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.226108 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.226243 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.229789 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.229798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.229806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.229851 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.233377 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.234044 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.234052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.234059 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.234223 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.234740 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.234744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.234747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.235117 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.235224 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.236031 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.236260 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 00:59:56.237917 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.219134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33921 (local address 127.0.97.2:40813)
0619 00:59:56.219782 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:56.219818 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:56.221410 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:56.222068 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:56.222075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:56.222545 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:56.223129 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:56.223145 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.224124 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.224129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:56.225311 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:56.225322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.226342 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.226347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:56.226540 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.227271 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:56.227291 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:56.229558 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:56.233495 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:56.233501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:56.233511 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:56.233874 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:56.234331 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:56.234339 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:56.234344 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:56.234573 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:56.235247 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:56.235257 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:56.235665 (+ 408us) client_negotiation.cc:769] Sending connection context
0619 00:59:56.235940 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 00:59:56.236477 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":272,"threads_started":1}
I20260619 00:59:56.239485 4305 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33921
I20260619 00:59:56.239779 4305 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:56.240447 4305 heartbeater.cc:511] Master 127.0.97.62:33921 requested a full tablet report, sending...
I20260619 00:59:56.241884 3982 ts_manager.cc:194] Registered new tserver with Master: 7a65e320b46043a6947a508ebce4cfdd (127.0.97.2:36009)
I20260619 00:59:56.242703 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40813
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 00:59:56.270272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:56.584594 4310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:56.584929 4310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:56.585094 4310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:56.592531 4310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:56.592662 4310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:56.592849 4310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 00:59:56.602582 4310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:56.603899 4310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:56.605207 4310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:56.618533 4318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:56.618625 4315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:56.618562 4316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:56.619282 4310 server_base.cc:1034] running on GCE node
I20260619 00:59:56.619874 4310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:56.620909 4310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:56.622154 4310 hybrid_clock.cc:648] HybridClock initialized: now 1781830796622087 us; error 66 us; skew 500 ppm
I20260619 00:59:56.622536 4310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:56.625674 4310 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 00:59:56.627147 4310 webserver.cc:460] Webserver started at http://127.0.97.3:40169/ using document root <none> and password file <none>
I20260619 00:59:56.627847 4310 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:56.627998 4310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:56.628309 4310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:56.631404 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "7eaa13ffa1e44da7bc02e84e4316f782"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "14e5d04a0d67f40c31fbd01d8b463453"
server_key_iv: "b1456be58a6f911b94b4a7bf6ae18d29"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.632325 4310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7eaa13ffa1e44da7bc02e84e4316f782"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "14e5d04a0d67f40c31fbd01d8b463453"
server_key_iv: "b1456be58a6f911b94b4a7bf6ae18d29"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.637286 4310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 00:59:56.640693 4325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:56.641878 4310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 00:59:56.642057 4310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "7eaa13ffa1e44da7bc02e84e4316f782"
format_stamp: "Formatted at 2026-06-19 00:59:56 on dist-test-slave-3rxd"
server_key: "14e5d04a0d67f40c31fbd01d8b463453"
server_key_iv: "b1456be58a6f911b94b4a7bf6ae18d29"
server_key_version: "encryptionkey@0"
I20260619 00:59:56.642261 4310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:56.675786 4310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:56.679672 4310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:56.679946 4310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:56.680850 4310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:56.682089 4310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:56.682190 4310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:56.682337 4310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:56.682410 4310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:56.715302 4310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43757
I20260619 00:59:56.715358 4437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43757 every 8 connection(s)
I20260619 00:59:56.716952 4310 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 00:59:56.717890 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4310
I20260619 00:59:56.718068 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 00:59:56.718550 388 external_mini_cluster.cc:1428] Setting key 3ecffa60274dde261bd1fa37a16c1e79
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:56.737427 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.721055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41071 (local address 127.0.97.62:33921)
0619 00:59:56.721226 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.721232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.722537 (+ 1305us) server_negotiation.cc:407] Connection header received
0619 00:59:56.724364 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.724369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.724449 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.724562 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.725936 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.726653 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.728174 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.728571 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.728704 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.731993 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.731997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.732000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.732031 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.734850 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.735519 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.735523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.735525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.735644 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.736039 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.736042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.736044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.736285 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.736390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.737049 (+ 659us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.737198 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 00:59:56.738323 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.721663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33921 (local address 127.0.97.3:41071)
0619 00:59:56.722384 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:56.722431 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:56.724121 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:56.724795 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:56.724803 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:56.725273 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:56.725773 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:56.725788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.726829 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.726835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:56.728032 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:56.728043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.728789 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.728798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:56.729031 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.729631 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:56.729655 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:56.731746 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:56.734993 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:56.734999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:56.735010 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:56.735391 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:56.735743 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:56.735746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:56.735748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:56.735937 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:56.736385 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:56.736390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:56.736700 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 00:59:56.736916 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 00:59:56.737246 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":304,"threads_started":1}
I20260619 00:59:56.739603 4438 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33921
I20260619 00:59:56.739895 4438 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:56.740566 4438 heartbeater.cc:511] Master 127.0.97.62:33921 requested a full tablet report, sending...
I20260619 00:59:56.741848 3982 ts_manager.cc:194] Registered new tserver with Master: 7eaa13ffa1e44da7bc02e84e4316f782 (127.0.97.3:43757)
I20260619 00:59:56.742632 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41071
I20260619 00:59:56.745918 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 00:59:56.752074 4171 heartbeater.cc:503] Master 127.0.97.62:33921 was elected leader, sending a full tablet report...
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 00:59:56.791879 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.777549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.0.97.1:43371)
0619 00:59:56.778170 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.778178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.778207 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:59:56.778271 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.778278 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.778500 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.778636 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.779931 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.780957 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.782161 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.782764 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.782863 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.786369 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.786379 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.786392 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.786432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.789351 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.789908 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.789915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.789918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.790032 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.790360 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.790364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.790366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.790695 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.790999 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.791243 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.791518 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:56.807801 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.795756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.97.62:33921)
0619 00:59:56.795955 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.795960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.796026 (+ 66us) server_negotiation.cc:407] Connection header received
0619 00:59:56.796160 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.796163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.796219 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.796291 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.797371 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.798139 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.799381 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.799753 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.799845 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.803223 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.803228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.803231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.803266 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.805654 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.806179 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.806182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.806184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.806286 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.806557 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.806560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.806562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.806898 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.807061 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.807324 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.807464 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 00:59:56.808360 3982 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58294
I20260619 00:59:56.809239 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4043
I20260619 00:59:56.820919 4445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.810010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.97.1:43371)
0619 00:59:56.810212 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.810220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.810449 (+ 229us) server_negotiation.cc:407] Connection header received
0619 00:59:56.810632 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.810638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.810702 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.810873 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.812025 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.812661 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.813796 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.814195 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.814315 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.816345 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.816351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.816355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.816391 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.818553 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.819169 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.819176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.819180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.819301 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.819652 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.819657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.819661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.819890 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.820149 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.820415 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.820614 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 00:59:56.821441 4166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:56.901896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4043
I20260619 00:59:56.917409 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4177
Jun 19 00:59:56 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 00:59:56.932752 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:56.918351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.0.97.2:36009)
0619 00:59:56.918910 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:56.918920 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:56.918946 (+ 26us) server_negotiation.cc:407] Connection header received
0619 00:59:56.919005 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:56.919012 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:56.919228 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:56.919371 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:56.920490 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.921489 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.922672 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:56.923284 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:56.923381 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:56.927284 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:56.927294 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:56.927308 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:56.927346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:56.930306 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.930969 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.930978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.930981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.931104 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:56.931401 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:56.931404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:56.931406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:56.931742 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:56.931948 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:56.932172 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 00:59:56.932377 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":255,"threads_started":1}
I20260619 00:59:56.933962 4300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:57.015501 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4177
I20260619 00:59:57.031368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4310
Jun 19 00:59:57 dist-test-slave-3rxd krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 00:59:57.048168 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:57.032302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.97.3:43757)
0619 00:59:57.032822 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:57.032828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:57.032857 (+ 29us) server_negotiation.cc:407] Connection header received
0619 00:59:57.032916 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:57.032922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:57.033156 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:57.033336 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:57.034592 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:57.035675 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:57.036957 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:57.037510 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:57.037643 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:57.042887 (+ 5244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:57.042898 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:57.042912 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:57.042944 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:57.045457 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:57.046186 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:57.046193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:57.046195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:57.046311 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:57.046620 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:57.046623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:57.046625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:57.047261 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:57.047508 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:57.047704 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 00:59:57.047887 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260619 00:59:57.049353 4433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:57.129232 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4310
I20260619 00:59:57.145052 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 3950
I20260619 00:59:57.146278 4012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 00:59:57.244002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 3950
2026-06-19T00:59:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3803 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 00:59:57 dist-test-slave-3rxd krb5kdc[4457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 00:59:57 dist-test-slave-3rxd krb5kdc[4457](info): set up 2 sockets
Jun 19 00:59:57 dist-test-slave-3rxd krb5kdc[4457](info): commencing operation
krb5kdc: starting...
W20260619 00:59:58.327291 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 00:59:58 dist-test-slave-3rxd krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T00:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T00:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 00:59:58.514070 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32975
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:58.828042 4470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:58.828361 4470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:58.828563 4470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:58.835961 4470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 00:59:58.836118 4470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:58.836215 4470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:58.836321 4470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 00:59:58.836416 4470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 00:59:58.846417 4470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:58.847779 4470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:58.849180 4470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:58.861078 4478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:58.861060 4475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:58.861378 4476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:58.861804 4470 server_base.cc:1034] running on GCE node
I20260619 00:59:58.862619 4470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:58.863983 4470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:58.865181 4470 hybrid_clock.cc:648] HybridClock initialized: now 1781830798865149 us; error 27 us; skew 500 ppm
I20260619 00:59:58.865525 4470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:58 dist-test-slave-3rxd krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:58.868549 4470 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 00:59:58.869648 4470 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:58.869735 4470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:58.870088 4470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:58.873070 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "6490048bfb0f452a9d0bd858638ec559"
format_stamp: "Formatted at 2026-06-19 00:59:58 on dist-test-slave-3rxd"
server_key: "2884b0fbf57685425926c49cce4bff75"
server_key_iv: "cc01797e740877b48578e86ba84c7208"
server_key_version: "encryptionkey@0"
I20260619 00:59:58.873827 4470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "6490048bfb0f452a9d0bd858638ec559"
format_stamp: "Formatted at 2026-06-19 00:59:58 on dist-test-slave-3rxd"
server_key: "2884b0fbf57685425926c49cce4bff75"
server_key_iv: "cc01797e740877b48578e86ba84c7208"
server_key_version: "encryptionkey@0"
I20260619 00:59:58.878584 4470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 00:59:58.881857 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:58.883185 4470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:58.883356 4470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "6490048bfb0f452a9d0bd858638ec559"
format_stamp: "Formatted at 2026-06-19 00:59:58 on dist-test-slave-3rxd"
server_key: "2884b0fbf57685425926c49cce4bff75"
server_key_iv: "cc01797e740877b48578e86ba84c7208"
server_key_version: "encryptionkey@0"
I20260619 00:59:58.883565 4470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:58.899593 4470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:58.903342 4470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:58.903609 4470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:58.921025 4470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32975
I20260619 00:59:58.921049 4535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32975 every 8 connection(s)
I20260619 00:59:58.921864 4470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 00:59:58.925292 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 00:59:58.931649 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4470
I20260619 00:59:58.931852 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 00:59:58.932466 388 external_mini_cluster.cc:1428] Setting key 02ae9ad1df5caf68730ceeb6e461d55f
I20260619 00:59:58.935321 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: Bootstrap starting.
I20260619 00:59:58.938571 4536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: Neither blocks nor log segments found. Creating new log.
I20260619 00:59:58.939733 4536 log.cc:824] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: Log is configured to *not* fsync() on all Append() calls
I20260619 00:59:58.942679 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: No bootstrap required, opened a new log
Jun 19 00:59:58 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:58.951581 4536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } }
I20260619 00:59:58.951953 4536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 00:59:58.952064 4536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6490048bfb0f452a9d0bd858638ec559, State: Initialized, Role: FOLLOWER
I20260619 00:59:58.952759 4536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } }
I20260619 00:59:58.953012 4536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 00:59:58.953162 4536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 00:59:58.953358 4536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 0 FOLLOWER]: Advancing to term 1
I20260619 00:59:58.955513 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:58.934263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.0.97.62:32975)
0619 00:59:58.935010 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:58.935021 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:58.935059 (+ 38us) server_negotiation.cc:407] Connection header received
0619 00:59:58.936709 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:58.936728 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:58.937092 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:58.937377 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:58.939231 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:58.940952 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:58.942819 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:58.943469 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:58.943590 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:58.948034 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:58.948042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:58.948056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:58.948101 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:58.951153 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:58.951819 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:58.951827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:58.951833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:58.952112 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:58.952546 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:58.952551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:58.952555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:58.953160 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:58.953469 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:58.953891 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 00:59:58.954279 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260619 00:59:58.956128 4536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } }
I20260619 00:59:58.956698 4536 leader_election.cc:302] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6490048bfb0f452a9d0bd858638ec559; no voters:
I20260619 00:59:58.957347 4536 leader_election.cc:288] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 00:59:58.957506 4541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 00:59:58.959775 4536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 00:59:58.964476 4541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [term 1 LEADER]: Becoming Leader. State: Replica: 6490048bfb0f452a9d0bd858638ec559, State: Running, Role: LEADER
I20260619 00:59:58.965137 4541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } }
I20260619 00:59:58.968117 4542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6490048bfb0f452a9d0bd858638ec559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } } }
I20260619 00:59:58.968479 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6490048bfb0f452a9d0bd858638ec559. Latest consensus state: current_term: 1 leader_uuid: "6490048bfb0f452a9d0bd858638ec559" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6490048bfb0f452a9d0bd858638ec559" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32975 } } }
I20260619 00:59:58.968703 4542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [sys.catalog]: This master's current role is: LEADER
I20260619 00:59:58.968775 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559 [sys.catalog]: This master's current role is: LEADER
W20260619 00:59:58.972901 4555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 00:59:58.973016 4555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 00:59:58.973155 4556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 00:59:58.974798 4556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 00:59:58.982254 4556 catalog_manager.cc:1347] Generated new cluster ID: f197dc74d59f4a69aea5b18a0f073408
I20260619 00:59:58.982357 4556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 00:59:59.005016 4556 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 00:59:59.006320 4556 catalog_manager.cc:1504] Loading token signing keys...
I20260619 00:59:59.018229 4556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: Generated new TSK 0
I20260619 00:59:59.019129 4556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 00:59:59.068783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:59.381201 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:59.381510 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:59.381670 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:59.388980 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:59.389119 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:59.389281 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 00:59:59.398877 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:59.400103 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:59.401356 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:59.414403 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:59.414490 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:59.414433 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:59.415283 4562 server_base.cc:1034] running on GCE node
I20260619 00:59:59.415817 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:59.416731 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:59.417924 4562 hybrid_clock.cc:648] HybridClock initialized: now 1781830799417868 us; error 47 us; skew 500 ppm
I20260619 00:59:59.418196 4562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:59 dist-test-slave-3rxd krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:59.421202 4562 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 00:59:59.422204 4562 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:59.422289 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:59.422536 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:59.425875 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "e69958ad83564897a9a55ea86d336573"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "c1571163027ae3089b7d12d67870767e"
server_key_iv: "fe5181dd789571826a7d9f6579a57ff3"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.427062 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e69958ad83564897a9a55ea86d336573"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "c1571163027ae3089b7d12d67870767e"
server_key_iv: "fe5181dd789571826a7d9f6579a57ff3"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.432008 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 00:59:59.435582 4576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.437444 4562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 00:59:59.437624 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "e69958ad83564897a9a55ea86d336573"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "c1571163027ae3089b7d12d67870767e"
server_key_iv: "fe5181dd789571826a7d9f6579a57ff3"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.437880 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:59.449864 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:59.453766 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:59.454028 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:59.454994 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:59.456661 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:59.456774 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.456902 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:59.457000 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.488926 4562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43867
I20260619 00:59:59.488976 4688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43867 every 8 connection(s)
I20260619 00:59:59.489840 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 00:59:59.497627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4562
I20260619 00:59:59.497836 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 00:59:59.498525 388 external_mini_cluster.cc:1428] Setting key eb7d3b492850c922b15738fc525a5c54
Jun 19 00:59:59 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:59.511472 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:59.492199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42363 (local address 127.0.97.62:32975)
0619 00:59:59.492610 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:59.492615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:59.493401 (+ 786us) server_negotiation.cc:407] Connection header received
0619 00:59:59.495263 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:59.495269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:59.495359 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:59.495483 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:59.496977 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.498058 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.499965 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.500376 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.500482 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:59.503873 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:59.503881 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:59.503889 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:59.503930 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:59.507026 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:59.508015 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:59.508023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:59.508028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:59.508191 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:59.508728 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:59.508736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:59.508741 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:59.509145 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:59.509307 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:59.510349 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0619 00:59:59.510776 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":217,"threads_started":1}
I20260619 00:59:59.512256 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:59.492620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32975 (local address 127.0.97.1:42363)
0619 00:59:59.493236 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:59.493279 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:59.494996 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:59.495712 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:59.495720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:59.496224 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:59.496808 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:59.496823 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.498222 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.498226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:59.499812 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:59.499825 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.500615 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.500621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:59.500803 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:59.501486 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:59.501519 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:59.503649 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:59.507153 (+ 3504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:59.507165 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:59.507186 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:59.507827 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:59.508329 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:59.508337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:59.508343 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:59.508594 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:59.509324 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:59.509336 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:59.509854 (+ 518us) client_negotiation.cc:769] Sending connection context
0619 00:59:59.510152 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 00:59:59.510761 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":246,"threads_started":1}
I20260619 00:59:59.514029 4689 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32975
I20260619 00:59:59.514348 4689 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:59.515228 4689 heartbeater.cc:511] Master 127.0.97.62:32975 requested a full tablet report, sending...
I20260619 00:59:59.517825 4501 ts_manager.cc:194] Registered new tserver with Master: e69958ad83564897a9a55ea86d336573 (127.0.97.1:43867)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 00:59:59.520052 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 00:59:59.545938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 00:59:59.854411 4695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 00:59:59.854686 4695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 00:59:59.854859 4695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 00:59:59.861987 4695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 00:59:59.862159 4695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 00:59:59.862349 4695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 00:59:59.871829 4695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 00:59:59.872988 4695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 00:59:59.874203 4695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 00:59:59.886029 4703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:59.886117 4700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 00:59:59.886147 4701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 00:59:59.887135 4695 server_base.cc:1034] running on GCE node
I20260619 00:59:59.887670 4695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 00:59:59.888586 4695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 00:59:59.889871 4695 hybrid_clock.cc:648] HybridClock initialized: now 1781830799889824 us; error 50 us; skew 500 ppm
I20260619 00:59:59.890204 4695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 00:59:59 dist-test-slave-3rxd krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 00:59:59.893162 4695 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 00:59:59.894279 4695 fs_manager.cc:359] Metadata directory not provided
I20260619 00:59:59.894402 4695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 00:59:59.894798 4695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 00:59:59.897505 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "7d646807a85d4cb396acdd9c7d7e0c7c"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "4cc6c5db6d2fe8d0f85692cdc9e98998"
server_key_iv: "ba7107cb772a4463fd5c90767fc93283"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.898288 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7d646807a85d4cb396acdd9c7d7e0c7c"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "4cc6c5db6d2fe8d0f85692cdc9e98998"
server_key_iv: "ba7107cb772a4463fd5c90767fc93283"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.902957 4695 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260619 00:59:59.906078 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.907505 4695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 00:59:59.907670 4695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "7d646807a85d4cb396acdd9c7d7e0c7c"
format_stamp: "Formatted at 2026-06-19 00:59:59 on dist-test-slave-3rxd"
server_key: "4cc6c5db6d2fe8d0f85692cdc9e98998"
server_key_iv: "ba7107cb772a4463fd5c90767fc93283"
server_key_version: "encryptionkey@0"
I20260619 00:59:59.907881 4695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 00:59:59.925529 4695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 00:59:59.930097 4695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 00:59:59.930335 4695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 00:59:59.931317 4695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 00:59:59.932752 4695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 00:59:59.932852 4695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.932945 4695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 00:59:59.933015 4695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 00:59:59.966418 4695 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42755
I20260619 00:59:59.966528 4821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42755 every 8 connection(s)
I20260619 00:59:59.967245 4695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 00:59:59.972981 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4695
I20260619 00:59:59.973189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 00:59:59.974071 388 external_mini_cluster.cc:1428] Setting key 66eceff14705c2fad27cb8e7e3c3a3b2
Jun 19 00:59:59 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830799, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 00:59:59.989054 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:59.969541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60429 (local address 127.0.97.62:32975)
0619 00:59:59.969760 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 00:59:59.969766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 00:59:59.970935 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 00:59:59.972607 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 00:59:59.972611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 00:59:59.972676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 00:59:59.972765 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 00:59:59.974877 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.976009 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.978434 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.978880 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.979008 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:59.983132 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 00:59:59.983137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 00:59:59.983140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 00:59:59.983170 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 00:59:59.985797 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:59.986500 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:59.986512 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:59.986519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:59.986630 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 00:59:59.987108 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 00:59:59.987116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 00:59:59.987122 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 00:59:59.987525 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 00:59:59.987698 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 00:59:59.988343 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 00:59:59.988604 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 00:59:59.989609 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 00:59:59.970045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32975 (local address 127.0.97.2:60429)
0619 00:59:59.970773 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0619 00:59:59.970821 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 00:59:59.972335 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 00:59:59.973069 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 00:59:59.973083 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 00:59:59.973813 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0619 00:59:59.974626 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:59.974664 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.976153 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.976159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:59.978287 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 00:59:59.978305 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 00:59:59.979123 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 00:59:59.979133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 00:59:59.979466 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 00:59:59.980162 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 00:59:59.980197 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 00:59:59.982914 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 00:59:59.985943 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:59.985953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:59.985968 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:59.986373 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:59.986768 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 00:59:59.986773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 00:59:59.986777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 00:59:59.986942 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 00:59:59.987645 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 00:59:59.987669 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 00:59:59.987977 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 00:59:59.988198 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 00:59:59.988558 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260619 00:59:59.991127 4822 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32975
I20260619 00:59:59.991468 4822 heartbeater.cc:464] Registering TS with master...
I20260619 00:59:59.992274 4822 heartbeater.cc:511] Master 127.0.97.62:32975 requested a full tablet report, sending...
I20260619 00:59:59.993803 4501 ts_manager.cc:194] Registered new tserver with Master: 7d646807a85d4cb396acdd9c7d7e0c7c (127.0.97.2:42755)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 00:59:59.994644 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:00:00.019497 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:00.333323 4827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:00.333598 4827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:00.333695 4827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:00.340971 4827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:00.341075 4827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:00.341233 4827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:00.350941 4827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:00.352152 4827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:00.353394 4827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:00.365870 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:00.365901 4833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:00.365950 4835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:00.366796 4827 server_base.cc:1034] running on GCE node
I20260619 01:00:00.367430 4827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:00.368278 4827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:00.369496 4827 hybrid_clock.cc:648] HybridClock initialized: now 1781830800369458 us; error 32 us; skew 500 ppm
I20260619 01:00:00.369812 4827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:00 dist-test-slave-3rxd krb5kdc[4457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:00.378360 4827 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:00:00.379556 4827 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:00.379644 4827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:00.380000 4827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:00.382977 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "95469b9fd1e640dd8de0d139a96edd20"
format_stamp: "Formatted at 2026-06-19 01:00:00 on dist-test-slave-3rxd"
server_key: "206e6f07bd247d346b939c405df17797"
server_key_iv: "fa155b732255d936a777ffc1c1466a7a"
server_key_version: "encryptionkey@0"
I20260619 01:00:00.383750 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "95469b9fd1e640dd8de0d139a96edd20"
format_stamp: "Formatted at 2026-06-19 01:00:00 on dist-test-slave-3rxd"
server_key: "206e6f07bd247d346b939c405df17797"
server_key_iv: "fa155b732255d936a777ffc1c1466a7a"
server_key_version: "encryptionkey@0"
I20260619 01:00:00.388849 4827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:00.392320 4841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:00.393563 4827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:00:00.393721 4827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "95469b9fd1e640dd8de0d139a96edd20"
format_stamp: "Formatted at 2026-06-19 01:00:00 on dist-test-slave-3rxd"
server_key: "206e6f07bd247d346b939c405df17797"
server_key_iv: "fa155b732255d936a777ffc1c1466a7a"
server_key_version: "encryptionkey@0"
I20260619 01:00:00.393910 4827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:00.414402 4827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:00.418349 4827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:00.418618 4827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:00.419572 4827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:00.420853 4827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:00.420936 4827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:00.421090 4827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:00.421176 4827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:00.452945 4827 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46131
I20260619 01:00:00.453037 4953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46131 every 8 connection(s)
I20260619 01:00:00.453799 4827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:00.456714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 4827
I20260619 01:00:00.456892 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:00.457315 388 external_mini_cluster.cc:1428] Setting key 0a44452d970e571e41b9b66a77db5dbd
Jun 19 01:00:00 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:00.473516 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.456251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39515 (local address 127.0.97.62:32975)
0619 01:00:00.456464 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.456471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.457773 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 01:00:00.459887 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.459891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.459956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.460089 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:00.461583 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.462272 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.463912 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.464392 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.464491 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.467784 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:00.467788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:00.467792 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:00.467822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:00.470469 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.471286 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.471294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.471302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.471414 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.471888 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.471891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.471893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.472281 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:00.472439 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.473101 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.473273 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:00:00.474442 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.456752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32975 (local address 127.0.97.3:39515)
0619 01:00:00.457596 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.457640 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.459595 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.460355 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.460367 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.460911 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:00.461447 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.461463 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.462393 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.462397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.463789 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.463800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.464572 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.464577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.464757 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.465403 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:00.465424 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:00.467599 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:00.470613 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:00.470621 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:00.470638 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:00.471157 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:00.471525 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:00.471531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:00.471537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:00.471773 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:00.472402 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:00.472408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:00.472737 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.472986 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.473377 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":372,"threads_started":1}
I20260619 01:00:00.475797 4954 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32975
I20260619 01:00:00.476054 4954 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:00.476639 4954 heartbeater.cc:511] Master 127.0.97.62:32975 requested a full tablet report, sending...
I20260619 01:00:00.477916 4501 ts_manager.cc:194] Registered new tserver with Master: 95469b9fd1e640dd8de0d139a96edd20 (127.0.97.3:46131)
I20260619 01:00:00.478753 4501 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39515
I20260619 01:00:00.484642 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:00:00.501780 4691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.489060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.0.97.62:32975)
0619 01:00:00.489264 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.489274 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.489815 (+ 541us) server_negotiation.cc:407] Connection header received
0619 01:00:00.490028 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.490032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.490090 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.490198 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:00.491836 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.492529 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.493701 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.494125 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.494257 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.496472 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:00.496478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:00.496482 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:00.496522 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:00.499184 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.499851 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.499859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.499862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.500019 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.500398 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.500404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.500407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.500685 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:00.500947 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.501254 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.501475 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:00:00.505601 4501 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:00.512441 4501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:00.523607 4689 heartbeater.cc:503] Master 127.0.97.62:32975 was elected leader, sending a full tablet report...
I20260619 01:00:00.545687 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.523693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.0.97.3:46131)
0619 01:00:00.535206 (+ 11513us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.535219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.536629 (+ 1410us) server_negotiation.cc:407] Connection header received
0619 01:00:00.537069 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.537082 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.537415 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.537601 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:00.538897 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.540535 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.543183 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.544584 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.544727 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.544981 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.545214 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.545373 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11276,"thread_start_us":11541,"threads_started":1}
I20260619 01:00:00.545727 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.523451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46131 (local address 127.0.0.1:39082)
0619 01:00:00.536482 (+ 13031us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.536522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.536774 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.538026 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.538036 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.538105 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:00.538618 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.538638 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.540684 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.540687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.543017 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.543030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.544776 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.544783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.544967 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.545012 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.545140 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.545315 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12776,"mutex_wait_us":2,"thread_start_us":15631,"threads_started":1}
I20260619 01:00:00.546630 4970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.523568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43867 (local address 127.0.0.1:55364)
0619 01:00:00.538927 (+ 15359us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.538949 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.539155 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.540016 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.540022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.540046 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:00.540451 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.540467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.542439 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.542443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.544065 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.544076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.545863 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.545870 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.545995 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.546022 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.546125 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.546237 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15247,"mutex_wait_us":6,"thread_start_us":12531,"threads_started":1}
I20260619 01:00:00.547118 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.524203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.0.97.1:43867)
0619 01:00:00.535915 (+ 11712us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.535927 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.539049 (+ 3122us) server_negotiation.cc:407] Connection header received
0619 01:00:00.539350 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.539362 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.539609 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.539755 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:00.540676 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.542290 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.544208 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.545409 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.545535 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.545731 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.546164 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.546304 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11521,"thread_start_us":11378,"threads_started":1}
I20260619 01:00:00.547470 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.524868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42755 (local address 127.0.0.1:53678)
0619 01:00:00.539339 (+ 14471us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.539353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.539499 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.539972 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.539976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.540008 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:00.540420 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.540429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.542462 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.542465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.544679 (+ 2214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.544691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.546640 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.546647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.546842 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.546872 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.546954 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.547086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14338,"mutex_wait_us":47,"thread_start_us":13782,"threads_started":1}
I20260619 01:00:00.548249 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.525493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.0.97.2:42755)
0619 01:00:00.538461 (+ 12968us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.538473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.539469 (+ 996us) server_negotiation.cc:407] Connection header received
0619 01:00:00.539646 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.539651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.539806 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.539985 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:00.540584 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.542290 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.544849 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.546198 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.546324 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.546498 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.547788 (+ 1290us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.547956 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12812,"thread_start_us":12715,"threads_started":1}
I20260619 01:00:00.550181 4624 tablet_service.cc:1467] Processing CreateTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 (DEFAULT_TABLE table=test-table [id=ab87798caae24f34b97abbd4f565aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:00.550256 4889 tablet_service.cc:1467] Processing CreateTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 (DEFAULT_TABLE table=test-table [id=ab87798caae24f34b97abbd4f565aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:00.551403 4624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c679cdff0084f79bfb86434fc5f7dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:00.551656 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c679cdff0084f79bfb86434fc5f7dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:00.551558 4757 tablet_service.cc:1467] Processing CreateTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 (DEFAULT_TABLE table=test-table [id=ab87798caae24f34b97abbd4f565aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:00.552783 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c679cdff0084f79bfb86434fc5f7dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:00.563635 4972 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Bootstrap starting.
I20260619 01:00:00.565806 4973 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Bootstrap starting.
I20260619 01:00:00.567263 4974 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Bootstrap starting.
I20260619 01:00:00.567523 4972 tablet_bootstrap.cc:654] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:00.568835 4972 log.cc:824] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:00.569199 4973 tablet_bootstrap.cc:654] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:00.569898 4974 tablet_bootstrap.cc:654] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:00.570477 4973 log.cc:824] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:00.571164 4974 log.cc:824] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:00.572584 4972 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: No bootstrap required, opened a new log
I20260619 01:00:00.572849 4972 ts_tablet_manager.cc:1397] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 01:00:00.573340 4973 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: No bootstrap required, opened a new log
I20260619 01:00:00.573593 4973 ts_tablet_manager.cc:1397] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 01:00:00.574960 4974 tablet_bootstrap.cc:492] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: No bootstrap required, opened a new log
I20260619 01:00:00.575184 4974 ts_tablet_manager.cc:1397] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260619 01:00:00.581687 4974 raft_consensus.cc:348] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.582077 4974 raft_consensus.cc:374] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:00.582193 4974 raft_consensus.cc:729] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d646807a85d4cb396acdd9c7d7e0c7c, State: Initialized, Role: FOLLOWER
I20260619 01:00:00.582834 4974 consensus_queue.cc:260] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.583799 4822 heartbeater.cc:503] Master 127.0.97.62:32975 was elected leader, sending a full tablet report...
I20260619 01:00:00.583593 4973 raft_consensus.cc:348] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.583593 4972 raft_consensus.cc:348] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.583968 4972 raft_consensus.cc:374] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:00.583971 4973 raft_consensus.cc:374] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:00.584085 4973 raft_consensus.cc:729] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95469b9fd1e640dd8de0d139a96edd20, State: Initialized, Role: FOLLOWER
I20260619 01:00:00.584085 4972 raft_consensus.cc:729] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69958ad83564897a9a55ea86d336573, State: Initialized, Role: FOLLOWER
I20260619 01:00:00.584637 4972 consensus_queue.cc:260] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.584637 4973 consensus_queue.cc:260] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.585945 4974 ts_tablet_manager.cc:1428] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260619 01:00:00.591396 4954 heartbeater.cc:503] Master 127.0.97.62:32975 was elected leader, sending a full tablet report...
I20260619 01:00:00.592128 4973 ts_tablet_manager.cc:1428] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Time spent starting tablet: real 0.018s user 0.010s sys 0.007s
I20260619 01:00:00.593442 4972 ts_tablet_manager.cc:1428] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
W20260619 01:00:00.705292 4955 tablet.cc:2369] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:00.711283 4979 raft_consensus.cc:482] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:00.711612 4979 raft_consensus.cc:504] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.713411 4979 leader_election.cc:288] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95469b9fd1e640dd8de0d139a96edd20 (127.0.97.3:46131), 7d646807a85d4cb396acdd9c7d7e0c7c (127.0.97.2:42755)
W20260619 01:00:00.718968 4823 tablet.cc:2369] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:00.722277 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.713917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60317 (local address 127.0.97.3:46131)
0619 01:00:00.714495 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.714502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.714762 (+ 260us) server_negotiation.cc:407] Connection header received
0619 01:00:00.715073 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.715080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.715160 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.715225 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:00.715998 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.717102 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.719899 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.721137 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.721267 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.721364 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.721878 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.722014 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
I20260619 01:00:00.722277 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.714605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55681 (local address 127.0.97.2:42755)
0619 01:00:00.714811 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.714817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.715387 (+ 570us) server_negotiation.cc:407] Connection header received
0619 01:00:00.715537 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.715540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.715598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.715672 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:00.716272 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.717304 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.719897 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.721137 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.721267 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.721363 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.721878 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.722011 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:00:00.722499 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.713865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42755 (local address 127.0.97.1:55681)
0619 01:00:00.715280 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.715301 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.715410 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.715723 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.715729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.715776 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:00.716116 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.716124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.717424 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.717429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.719734 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.719748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.721420 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.721428 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.721680 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.721744 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.721836 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.722031 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1306,"mutex_wait_us":209,"thread_start_us":1093,"threads_started":1}
I20260619 01:00:00.722558 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.713593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46131 (local address 127.0.97.1:60317)
0619 01:00:00.714630 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:00.714651 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:00.714885 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:00.715417 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:00.715421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:00.715445 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:00.715823 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.715835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.717225 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.717229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.719734 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:00.719748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.721435 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.721439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:00.721664 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.721739 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 01:00:00.721836 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:00:00.722020 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":112,"thread_start_us":695,"threads_started":1}
I20260619 01:00:00.723551 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c679cdff0084f79bfb86434fc5f7dc0" candidate_uuid: "e69958ad83564897a9a55ea86d336573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95469b9fd1e640dd8de0d139a96edd20" is_pre_election: true
I20260619 01:00:00.723546 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c679cdff0084f79bfb86434fc5f7dc0" candidate_uuid: "e69958ad83564897a9a55ea86d336573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" is_pre_election: true
I20260619 01:00:00.724053 4777 raft_consensus.cc:2454] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69958ad83564897a9a55ea86d336573 in term 0.
I20260619 01:00:00.724046 4909 raft_consensus.cc:2454] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e69958ad83564897a9a55ea86d336573 in term 0.
I20260619 01:00:00.724677 4580 leader_election.cc:302] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d646807a85d4cb396acdd9c7d7e0c7c, e69958ad83564897a9a55ea86d336573; no voters:
I20260619 01:00:00.725118 4979 raft_consensus.cc:2781] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:00.725247 4979 raft_consensus.cc:482] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:00.725368 4979 raft_consensus.cc:3037] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:00.727648 4979 raft_consensus.cc:504] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.728286 4979 leader_election.cc:288] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [CANDIDATE]: Term 1 election: Requested vote from peers 95469b9fd1e640dd8de0d139a96edd20 (127.0.97.3:46131), 7d646807a85d4cb396acdd9c7d7e0c7c (127.0.97.2:42755)
I20260619 01:00:00.728725 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c679cdff0084f79bfb86434fc5f7dc0" candidate_uuid: "e69958ad83564897a9a55ea86d336573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95469b9fd1e640dd8de0d139a96edd20"
I20260619 01:00:00.728782 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c679cdff0084f79bfb86434fc5f7dc0" candidate_uuid: "e69958ad83564897a9a55ea86d336573" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c"
I20260619 01:00:00.728912 4909 raft_consensus.cc:3037] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:00.728987 4777 raft_consensus.cc:3037] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:00.731148 4909 raft_consensus.cc:2454] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69958ad83564897a9a55ea86d336573 in term 1.
I20260619 01:00:00.731168 4777 raft_consensus.cc:2454] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e69958ad83564897a9a55ea86d336573 in term 1.
I20260619 01:00:00.731547 4577 leader_election.cc:302] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95469b9fd1e640dd8de0d139a96edd20, e69958ad83564897a9a55ea86d336573; no voters:
I20260619 01:00:00.731882 4979 raft_consensus.cc:2781] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:00.732359 4979 raft_consensus.cc:686] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 1 LEADER]: Becoming Leader. State: Replica: e69958ad83564897a9a55ea86d336573, State: Running, Role: LEADER
I20260619 01:00:00.732832 4979 consensus_queue.cc:237] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } }
I20260619 01:00:00.737565 4500 catalog_manager.cc:5495] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 reported cstate change: term changed from 0 to 1, leader changed from <none> to e69958ad83564897a9a55ea86d336573 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e69958ad83564897a9a55ea86d336573" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e69958ad83564897a9a55ea86d336573" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 } health_report { overall_health: UNKNOWN } } }
W20260619 01:00:00.742228 4690 tablet.cc:2369] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:00.772214 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.767376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.0.97.1:43867)
0619 01:00:00.767593 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.767600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.767653 (+ 53us) server_negotiation.cc:407] Connection header received
0619 01:00:00.767814 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.767818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.767889 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.768010 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:00.768310 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.769026 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.770316 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.770696 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.770848 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.771233 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:00.771697 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:00.771803 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.771862 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.771917 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:00:00.779904 4777 raft_consensus.cc:1261] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 1 FOLLOWER]: Refusing update from remote peer e69958ad83564897a9a55ea86d336573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:00.779963 4909 raft_consensus.cc:1261] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 1 FOLLOWER]: Refusing update from remote peer e69958ad83564897a9a55ea86d336573: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:00.780864 4983 consensus_queue.cc:1035] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95469b9fd1e640dd8de0d139a96edd20" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:00.781239 4979 consensus_queue.cc:1035] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d646807a85d4cb396acdd9c7d7e0c7c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:00.791878 4986 mvcc.cc:205] Tried to move back new op lower bound from 7298378959985459200 to 7298378959805132800. Current Snapshot: MvccSnapshot[applied={T|T < 7298378959985459200}]
I20260619 01:00:00.805254 4500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:00.805521 4500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:00.808766 4500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6490048bfb0f452a9d0bd858638ec559: Sending DeleteTablet for 3 replicas of tablet 8c679cdff0084f79bfb86434fc5f7dc0
I20260619 01:00:00.809890 4624 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55364
I20260619 01:00:00.809974 4757 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53678
I20260619 01:00:00.809981 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c679cdff0084f79bfb86434fc5f7dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39082
I20260619 01:00:00.817575 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4562
I20260619 01:00:00.822286 4996 tablet_replica.cc:330] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: stopping tablet replica
I20260619 01:00:00.822676 4996 raft_consensus.cc:2229] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:00.823256 4996 raft_consensus.cc:2258] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:00.823807 4997 tablet_replica.cc:330] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: stopping tablet replica
I20260619 01:00:00.824338 4997 raft_consensus.cc:2229] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:00.825304 4997 raft_consensus.cc:2258] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:00.825736 4996 ts_tablet_manager.cc:1905] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:00.828333 4995 tablet_replica.cc:330] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: stopping tablet replica
I20260619 01:00:00.828915 4995 raft_consensus.cc:2229] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:00.829701 4995 raft_consensus.cc:2258] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:00.831775 4995 ts_tablet_manager.cc:1905] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:00.831913 4997 ts_tablet_manager.cc:1905] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:00.833336 4996 ts_tablet_manager.cc:1918] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:00.833500 4996 log.cc:1196] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/8c679cdff0084f79bfb86434fc5f7dc0
I20260619 01:00:00.834045 4996 ts_tablet_manager.cc:1939] T 8c679cdff0084f79bfb86434fc5f7dc0 P 7d646807a85d4cb396acdd9c7d7e0c7c: Deleting consensus metadata
I20260619 01:00:00.835871 4488 catalog_manager.cc:4841] TS 7d646807a85d4cb396acdd9c7d7e0c7c (127.0.97.2:42755): tablet 8c679cdff0084f79bfb86434fc5f7dc0 (table test-table [id=ab87798caae24f34b97abbd4f565aad8]) successfully deleted
I20260619 01:00:00.838287 4995 ts_tablet_manager.cc:1918] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:00.838486 4995 log.cc:1196] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/8c679cdff0084f79bfb86434fc5f7dc0
I20260619 01:00:00.838662 4997 ts_tablet_manager.cc:1918] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:00.838851 4997 log.cc:1196] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/8c679cdff0084f79bfb86434fc5f7dc0
I20260619 01:00:00.839110 4995 ts_tablet_manager.cc:1939] T 8c679cdff0084f79bfb86434fc5f7dc0 P e69958ad83564897a9a55ea86d336573: Deleting consensus metadata
I20260619 01:00:00.839363 4997 ts_tablet_manager.cc:1939] T 8c679cdff0084f79bfb86434fc5f7dc0 P 95469b9fd1e640dd8de0d139a96edd20: Deleting consensus metadata
I20260619 01:00:00.841029 4487 catalog_manager.cc:4841] TS e69958ad83564897a9a55ea86d336573 (127.0.97.1:43867): tablet 8c679cdff0084f79bfb86434fc5f7dc0 (table test-table [id=ab87798caae24f34b97abbd4f565aad8]) successfully deleted
I20260619 01:00:00.841058 4485 catalog_manager.cc:4841] TS 95469b9fd1e640dd8de0d139a96edd20 (127.0.97.3:46131): tablet 8c679cdff0084f79bfb86434fc5f7dc0 (table test-table [id=ab87798caae24f34b97abbd4f565aad8]) successfully deleted
Jun 19 01:00:00 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:00.851073 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.818993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.97.1:43867)
0619 01:00:00.819168 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.819174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.829334 (+ 10160us) server_negotiation.cc:407] Connection header received
0619 01:00:00.829807 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.829814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.829907 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.829996 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:00.831733 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.832781 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.834570 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.835283 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.835385 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.843675 (+ 8290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:00.843693 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:00.843718 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:00.843794 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:00.847810 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.848603 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.848614 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.848618 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.848794 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.849260 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.849266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.849270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.849768 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:00.850108 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.850384 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.850639 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:00.851887 4684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:00.947854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4562
I20260619 01:00:00.966562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4695
Jun 19 01:00:00 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:00:00.982254 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:00.967491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.0.97.2:42755)
0619 01:00:00.967692 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:00.967696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:00.967710 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:00:00.967757 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:00.967760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:00.967838 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:00.967957 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:00.969123 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.969987 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.971385 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:00.971790 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:00.971918 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:00.975781 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:00.975792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:00.975809 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:00.975845 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:00.979606 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.980247 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.980256 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.980260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.980394 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:00.980762 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:00.980766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:00.980769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:00.981222 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:00.981481 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:00.981703 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 01:00:00.981897 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:00:00.982931 4817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:01.070348 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4695
I20260619 01:00:01.087217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4827
Jun 19 01:00:01 dist-test-slave-3rxd krb5kdc[4457](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:00:01.101473 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:01.087950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39084 (local address 127.0.97.3:46131)
0619 01:00:01.088163 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:01.088169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:01.088187 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:00:01.088353 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:01.088359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:01.088434 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:01.088538 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:01.089781 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:01.090496 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:01.091774 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:01.092154 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:01.092281 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:01.096259 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:01.096278 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:01.096293 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:01.096322 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:01.098881 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:01.099489 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:01.099498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:01.099502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:01.099662 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:01.100023 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:01.100028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:01.100031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:01.100372 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:01.100690 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:01.100958 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 01:00:01.101169 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 01:00:01.102082 4949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:01.189337 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4827
I20260619 01:00:01.205443 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 4470
I20260619 01:00:01.206519 4531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:01.310827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 4470
2026-06-19T01:00:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4070 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T01:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:01Z Disabled control of system clock
I20260619 01:00:01.376065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43125
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 01:00:01.689262 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:01.689566 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:01.689754 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:01.697186 5006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:01.697299 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:01.697386 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:01.697470 5006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:01.697551 5006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:01.706691 5006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43125
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:01.708081 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:01.709856 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:01.721570 5011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:01.721486 5012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:01.721516 5014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:01.722285 5006 server_base.cc:1034] running on GCE node
I20260619 01:00:01.723167 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:01.724476 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:01.725687 5006 hybrid_clock.cc:648] HybridClock initialized: now 1781830801725623 us; error 54 us; skew 500 ppm
I20260619 01:00:01.725997 5006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:01.728255 5006 webserver.cc:460] Webserver started at http://127.0.97.62:46625/ using document root <none> and password file <none>
I20260619 01:00:01.728906 5006 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:01.728992 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:01.729254 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:01.732121 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "1b4b1c1a4f924de8b7588a7395f91b16"
format_stamp: "Formatted at 2026-06-19 01:00:01 on dist-test-slave-3rxd"
server_key: "09a85873c8f2b9f5633a1747e82c31ff"
server_key_iv: "0ff91a236c16b3cf818db9a1bf527a56"
server_key_version: "encryptionkey@0"
I20260619 01:00:01.732856 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b4b1c1a4f924de8b7588a7395f91b16"
format_stamp: "Formatted at 2026-06-19 01:00:01 on dist-test-slave-3rxd"
server_key: "09a85873c8f2b9f5633a1747e82c31ff"
server_key_iv: "0ff91a236c16b3cf818db9a1bf527a56"
server_key_version: "encryptionkey@0"
I20260619 01:00:01.737669 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:01.740947 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:01.742260 5006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:00:01.742391 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "1b4b1c1a4f924de8b7588a7395f91b16"
format_stamp: "Formatted at 2026-06-19 01:00:01 on dist-test-slave-3rxd"
server_key: "09a85873c8f2b9f5633a1747e82c31ff"
server_key_iv: "0ff91a236c16b3cf818db9a1bf527a56"
server_key_version: "encryptionkey@0"
I20260619 01:00:01.742585 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:01.748020 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:01.748713 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:01.748951 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:01.766758 5006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43125
I20260619 01:00:01.766849 5071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43125 every 8 connection(s)
I20260619 01:00:01.768066 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:01.771689 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:01.772872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5006
I20260619 01:00:01.773074 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:01.773911 388 external_mini_cluster.cc:1428] Setting key 23827259e2d893df49103d6dc2061bd5
I20260619 01:00:01.781984 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:01.776033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.0.97.62:43125)
0619 01:00:01.777014 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:01.777025 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:01.777066 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:00:01.778515 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:01.778540 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:01.779042 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:01.779424 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:01.780421 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:01.780427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:01.780439 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:01.780454 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:01.780514 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:01.780529 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:01.780744 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:01.780868 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 01:00:01.781147 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260619 01:00:01.784631 5072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: Bootstrap starting.
I20260619 01:00:01.787448 5072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:01.788403 5072 log.cc:824] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:01.790886 5072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: No bootstrap required, opened a new log
I20260619 01:00:01.797451 5072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } }
I20260619 01:00:01.797796 5072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:01.797926 5072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b4b1c1a4f924de8b7588a7395f91b16, State: Initialized, Role: FOLLOWER
I20260619 01:00:01.798483 5072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } }
I20260619 01:00:01.798702 5072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:01.798897 5072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:01.799068 5072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:01.801067 5072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } }
I20260619 01:00:01.801538 5072 leader_election.cc:302] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b4b1c1a4f924de8b7588a7395f91b16; no voters:
I20260619 01:00:01.802068 5072 leader_election.cc:288] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:01.802235 5077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:01.803082 5077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [term 1 LEADER]: Becoming Leader. State: Replica: 1b4b1c1a4f924de8b7588a7395f91b16, State: Running, Role: LEADER
I20260619 01:00:01.803614 5077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } }
I20260619 01:00:01.804133 5072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:01.806602 5078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } } }
I20260619 01:00:01.807070 5078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:01.806928 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b4b1c1a4f924de8b7588a7395f91b16. Latest consensus state: current_term: 1 leader_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b4b1c1a4f924de8b7588a7395f91b16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43125 } } }
I20260619 01:00:01.807299 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:01.810458 5084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:01.815454 5084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:01.823956 5084 catalog_manager.cc:1347] Generated new cluster ID: 268398724fb5418489ef375010e53316
I20260619 01:00:01.824081 5084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:01.842115 5084 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:01.842494 5084 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:01.852294 5084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: Generated new TSK 0
I20260619 01:00:01.853092 5084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:01.867864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--builtin_ntp_servers=127.0.97.20:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 01:00:02.194228 5096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:02.194509 5096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:02.194597 5096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:02.201751 5096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:02.201853 5096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:02.201962 5096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:02.211135 5096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:02.212558 5096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:02.214237 5096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:02.227649 5102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:02.227680 5104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:02.227715 5101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:02.228377 5096 server_base.cc:1034] running on GCE node
I20260619 01:00:02.228916 5096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:02.229811 5096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:02.231003 5096 hybrid_clock.cc:648] HybridClock initialized: now 1781830802230943 us; error 63 us; skew 500 ppm
I20260619 01:00:02.231328 5096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:02.233580 5096 webserver.cc:460] Webserver started at http://127.0.97.1:40339/ using document root <none> and password file <none>
I20260619 01:00:02.234265 5096 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:02.234370 5096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:02.234687 5096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:02.237563 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "bdc252ce63da471d90cdaa58eb1b3631"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "b8855aef061f81c2a13ef697cac17218"
server_key_iv: "9b8069fa723e8d420f67fe53e1342658"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.238348 5096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bdc252ce63da471d90cdaa58eb1b3631"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "b8855aef061f81c2a13ef697cac17218"
server_key_iv: "9b8069fa723e8d420f67fe53e1342658"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.243414 5096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:00:02.247045 5110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.248364 5096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:00:02.248536 5096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "bdc252ce63da471d90cdaa58eb1b3631"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "b8855aef061f81c2a13ef697cac17218"
server_key_iv: "9b8069fa723e8d420f67fe53e1342658"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.248762 5096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:02.254418 5096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:02.255290 5096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:02.255558 5096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:02.256587 5096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:02.257977 5096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:02.258060 5096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.258205 5096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:02.258299 5096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.291159 5096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46653
I20260619 01:00:02.291211 5222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46653 every 8 connection(s)
I20260619 01:00:02.292474 5096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:02.297390 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5096
I20260619 01:00:02.297565 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:02.298193 388 external_mini_cluster.cc:1428] Setting key 92af70c52c35abe88b14dcbde0eb5832
I20260619 01:00:02.300806 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:02.294965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55317 (local address 127.0.97.62:43125)
0619 01:00:02.295375 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:02.295380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:02.296610 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 01:00:02.298377 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:02.298382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:02.298450 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:02.298573 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:02.300004 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:02.300010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:02.300014 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:02.300032 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:02.300079 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:02.300096 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:02.300192 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:02.300465 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 01:00:02.300521 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":211,"threads_started":1}
I20260619 01:00:02.301259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--builtin_ntp_servers=127.0.97.20:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 01:00:02.301771 5226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:02.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43125 (local address 127.0.97.1:55317)
0619 01:00:02.296443 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:02.296489 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:02.298081 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:02.298858 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:02.298870 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:02.298996 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:02.299673 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:02.299677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:02.299724 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:02.299829 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:02.300204 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:02.300215 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:02.300223 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 01:00:02.300509 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 01:00:02.300699 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":259,"threads_started":1}
I20260619 01:00:02.303284 5223 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43125
I20260619 01:00:02.303563 5223 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:02.304123 5223 heartbeater.cc:511] Master 127.0.97.62:43125 requested a full tablet report, sending...
I20260619 01:00:02.306382 5037 ts_manager.cc:194] Registered new tserver with Master: bdc252ce63da471d90cdaa58eb1b3631 (127.0.97.1:46653)
W20260619 01:00:02.631443 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:02.631783 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:02.631953 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:02.639293 5227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:02.639428 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:02.639606 5227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:02.649456 5227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:02.650848 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:02.652648 5227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:02.665424 5233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:02.665463 5235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:02.665380 5232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:02.666145 5227 server_base.cc:1034] running on GCE node
I20260619 01:00:02.666791 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:02.667762 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:02.669032 5227 hybrid_clock.cc:648] HybridClock initialized: now 1781830802668986 us; error 42 us; skew 500 ppm
I20260619 01:00:02.669374 5227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:02.671736 5227 webserver.cc:460] Webserver started at http://127.0.97.2:35463/ using document root <none> and password file <none>
I20260619 01:00:02.672421 5227 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:02.672541 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:02.672856 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:02.675678 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "83962273cff243109eb68f737c612a64"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "cb28a9482d743613c43b1ab0d53e2d18"
server_key_iv: "e5c3f20f0f36bcc81070b243718a9853"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.676468 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "83962273cff243109eb68f737c612a64"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "cb28a9482d743613c43b1ab0d53e2d18"
server_key_iv: "e5c3f20f0f36bcc81070b243718a9853"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.681320 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 01:00:02.684600 5241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.685885 5227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:00:02.686080 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "83962273cff243109eb68f737c612a64"
format_stamp: "Formatted at 2026-06-19 01:00:02 on dist-test-slave-3rxd"
server_key: "cb28a9482d743613c43b1ab0d53e2d18"
server_key_iv: "e5c3f20f0f36bcc81070b243718a9853"
server_key_version: "encryptionkey@0"
I20260619 01:00:02.686297 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:02.691872 5227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:02.692641 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:02.692902 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:02.693791 5227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:02.695217 5227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:02.695338 5227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.695470 5227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:02.695560 5227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:02.728729 5227 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40229
I20260619 01:00:02.728816 5353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40229 every 8 connection(s)
I20260619 01:00:02.730053 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:02.737373 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5227
I20260619 01:00:02.737571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:02.737896 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:02.732590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36335 (local address 127.0.97.62:43125)
0619 01:00:02.732804 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:02.732810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:02.734262 (+ 1452us) server_negotiation.cc:407] Connection header received
0619 01:00:02.735845 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:02.735849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:02.735897 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:02.735984 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:02.737193 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:02.737201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:02.737206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:02.737223 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:02.737256 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:02.737270 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:02.737392 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:02.737565 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 01:00:02.737628 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 01:00:02.738081 388 external_mini_cluster.cc:1428] Setting key e1028362075e1c39ee11309aff140732
I20260619 01:00:02.738934 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:02.733182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43125 (local address 127.0.97.2:36335)
0619 01:00:02.734097 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:02.734132 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:02.735603 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:02.736185 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:02.736193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:02.736278 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:02.736871 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:02.736874 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:02.736912 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:02.737025 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:02.737374 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:02.737381 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:02.737386 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:02.737629 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 01:00:02.737853 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":285,"threads_started":1}
I20260619 01:00:02.740381 5354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43125
I20260619 01:00:02.740666 5354 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:02.740947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--builtin_ntp_servers=127.0.97.20:41463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 01:00:02.741247 5354 heartbeater.cc:511] Master 127.0.97.62:43125 requested a full tablet report, sending...
I20260619 01:00:02.742642 5036 ts_manager.cc:194] Registered new tserver with Master: 83962273cff243109eb68f737c612a64 (127.0.97.2:40229)
W20260619 01:00:03.069299 5357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:03.069576 5357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:03.069686 5357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:03.077057 5357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:03.077167 5357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:03.077275 5357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:03.086253 5357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43125
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:03.087548 5357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:03.089272 5357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:03.102300 5362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:03.102183 5365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:03.102422 5363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:03.103262 5357 server_base.cc:1034] running on GCE node
I20260619 01:00:03.103947 5357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:03.105010 5357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:03.106242 5357 hybrid_clock.cc:648] HybridClock initialized: now 1781830803106209 us; error 30 us; skew 500 ppm
I20260619 01:00:03.106578 5357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:03.108903 5357 webserver.cc:460] Webserver started at http://127.0.97.3:36621/ using document root <none> and password file <none>
I20260619 01:00:03.109714 5357 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:03.109844 5357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:03.110172 5357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:03.113214 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "f15f702a75094995abce1a4f505d9673"
format_stamp: "Formatted at 2026-06-19 01:00:03 on dist-test-slave-3rxd"
server_key: "194a15a23eed9bd5734fb7325439f90d"
server_key_iv: "934ddef138fa1588aabe8fa3dbd85236"
server_key_version: "encryptionkey@0"
I20260619 01:00:03.114079 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f15f702a75094995abce1a4f505d9673"
format_stamp: "Formatted at 2026-06-19 01:00:03 on dist-test-slave-3rxd"
server_key: "194a15a23eed9bd5734fb7325439f90d"
server_key_iv: "934ddef138fa1588aabe8fa3dbd85236"
server_key_version: "encryptionkey@0"
I20260619 01:00:03.119324 5357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 01:00:03.123071 5371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:03.124507 5357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:03.124697 5357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "f15f702a75094995abce1a4f505d9673"
format_stamp: "Formatted at 2026-06-19 01:00:03 on dist-test-slave-3rxd"
server_key: "194a15a23eed9bd5734fb7325439f90d"
server_key_iv: "934ddef138fa1588aabe8fa3dbd85236"
server_key_version: "encryptionkey@0"
I20260619 01:00:03.124938 5357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:03.130635 5357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:03.131428 5357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:03.131688 5357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:03.132566 5357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:03.133906 5357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:03.134029 5357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:03.134150 5357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:03.134236 5357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:03.167233 5357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38841
I20260619 01:00:03.167322 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38841 every 8 connection(s)
I20260619 01:00:03.168572 5357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:03.170819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5357
I20260619 01:00:03.170996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:03.171478 388 external_mini_cluster.cc:1428] Setting key 33603f8814c7b1ff59659d187e13d327
I20260619 01:00:03.176586 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.172407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60407 (local address 127.0.97.62:43125)
0619 01:00:03.172583 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.172588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.173105 (+ 517us) server_negotiation.cc:407] Connection header received
0619 01:00:03.174681 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.174684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.174779 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.174868 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.176023 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.176027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.176030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.176044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.176073 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.176084 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.176179 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.176333 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.176377 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:00:03.177510 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.171890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43125 (local address 127.0.97.3:60407)
0619 01:00:03.172939 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.172989 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.174445 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.175072 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.175081 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.175172 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.175763 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.175766 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.175801 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.175882 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.176168 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.176175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.176180 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.176381 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.176578 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":316,"threads_started":1}
I20260619 01:00:03.178861 5484 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43125
I20260619 01:00:03.179150 5484 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:03.179708 5484 heartbeater.cc:511] Master 127.0.97.62:43125 requested a full tablet report, sending...
I20260619 01:00:03.180979 5036 ts_manager.cc:194] Registered new tserver with Master: f15f702a75094995abce1a4f505d9673 (127.0.97.3:38841)
I20260619 01:00:03.186643 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:00:03.193460 5225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.190916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.0.97.62:43125)
0619 01:00:03.191078 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.191100 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.191553 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:00:03.191754 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.191759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.191822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.191919 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.192800 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.192806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.192811 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.192828 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.192868 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.192880 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.193001 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.193139 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.193197 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 01:00:03.197229 5036 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:03.203670 5036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:03.221391 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.215999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38841 (local address 127.0.0.1:37604)
0619 01:00:03.216717 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.216756 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.216871 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.218813 (+ 1942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.218823 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.218938 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.219829 (+ 891us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.219835 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.219881 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.219969 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.220543 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.220555 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.220567 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.220753 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.220851 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":369,"threads_started":1}
I20260619 01:00:03.222208 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.216191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37604 (local address 127.0.97.3:38841)
0619 01:00:03.216897 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.216905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.216960 (+ 55us) server_negotiation.cc:407] Connection header received
0619 01:00:03.217026 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.217036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.217270 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.217436 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.220171 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.220182 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.220202 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.220223 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.220286 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.220309 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.220489 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.221660 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.221771 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":410,"threads_started":1}
I20260619 01:00:03.226200 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.217470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40229 (local address 127.0.0.1:54594)
0619 01:00:03.223815 (+ 6345us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.223828 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.223951 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.224512 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.224517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.224539 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.225236 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.225239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.225255 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.225295 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.225637 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.225642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.225646 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.225765 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.225811 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6171,"thread_start_us":6031,"threads_started":1}
I20260619 01:00:03.226197 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.218044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.97.2:40229)
0619 01:00:03.223311 (+ 5267us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.223323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.223954 (+ 631us) server_negotiation.cc:407] Connection header received
0619 01:00:03.224065 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.224076 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.224365 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.224525 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.225466 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.225472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.225481 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.225493 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.225530 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.225546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.225708 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.225769 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.225862 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5083,"thread_start_us":4967,"threads_started":1}
I20260619 01:00:03.228780 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 (DEFAULT_TABLE table=test-table [id=d9fc4245fa0348198b84759992791e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:03.229596 5289 tablet_service.cc:1467] Processing CreateTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 (DEFAULT_TABLE table=test-table [id=d9fc4245fa0348198b84759992791e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:03.230144 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c7e01831034fb3b1319bc2fe4dce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:03.230854 5289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c7e01831034fb3b1319bc2fe4dce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:03.232633 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.217148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45880)
0619 01:00:03.222556 (+ 5408us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.222578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.222667 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.230524 (+ 7857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.230529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.230563 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.231179 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.231183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.231200 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.231255 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.232248 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.232252 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.232258 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.232340 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.232393 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5284,"thread_start_us":7012,"threads_started":1}
I20260619 01:00:03.233462 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.217403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.0.97.1:46653)
0619 01:00:03.227087 (+ 9684us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.227103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.227130 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:03.227196 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.227204 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.230326 (+ 3122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.230485 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.231610 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.231619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.231632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.231649 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.231703 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.231723 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.232988 (+ 1265us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.233064 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.233155 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9535,"thread_start_us":9941,"threads_started":1}
I20260619 01:00:03.236812 5158 tablet_service.cc:1467] Processing CreateTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 (DEFAULT_TABLE table=test-table [id=d9fc4245fa0348198b84759992791e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:03.238061 5158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c7e01831034fb3b1319bc2fe4dce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:03.242084 5502 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Bootstrap starting.
I20260619 01:00:03.244108 5503 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Bootstrap starting.
I20260619 01:00:03.246513 5502 tablet_bootstrap.cc:654] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:03.247464 5503 tablet_bootstrap.cc:654] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:03.247805 5502 log.cc:824] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:03.248651 5503 log.cc:824] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:03.251152 5506 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Bootstrap starting.
I20260619 01:00:03.251523 5502 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: No bootstrap required, opened a new log
I20260619 01:00:03.251869 5502 ts_tablet_manager.cc:1397] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 01:00:03.255144 5506 tablet_bootstrap.cc:654] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:03.256198 5506 log.cc:824] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:03.257438 5503 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: No bootstrap required, opened a new log
I20260619 01:00:03.257720 5503 ts_tablet_manager.cc:1397] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260619 01:00:03.258651 5506 tablet_bootstrap.cc:492] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: No bootstrap required, opened a new log
I20260619 01:00:03.258937 5506 ts_tablet_manager.cc:1397] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 01:00:03.263537 5502 raft_consensus.cc:348] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.264205 5502 raft_consensus.cc:374] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:03.264353 5502 raft_consensus.cc:729] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15f702a75094995abce1a4f505d9673, State: Initialized, Role: FOLLOWER
I20260619 01:00:03.265177 5502 consensus_queue.cc:260] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.266340 5484 heartbeater.cc:503] Master 127.0.97.62:43125 was elected leader, sending a full tablet report...
I20260619 01:00:03.267231 5502 ts_tablet_manager.cc:1428] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260619 01:00:03.267341 5506 raft_consensus.cc:348] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.267804 5506 raft_consensus.cc:374] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:03.267941 5506 raft_consensus.cc:729] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdc252ce63da471d90cdaa58eb1b3631, State: Initialized, Role: FOLLOWER
I20260619 01:00:03.268594 5506 consensus_queue.cc:260] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.268579 5503 raft_consensus.cc:348] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.268994 5503 raft_consensus.cc:374] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:03.269147 5503 raft_consensus.cc:729] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83962273cff243109eb68f737c612a64, State: Initialized, Role: FOLLOWER
I20260619 01:00:03.269953 5503 consensus_queue.cc:260] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.271273 5354 heartbeater.cc:503] Master 127.0.97.62:43125 was elected leader, sending a full tablet report...
I20260619 01:00:03.272061 5503 ts_tablet_manager.cc:1428] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260619 01:00:03.275238 5509 raft_consensus.cc:482] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:03.275318 5223 heartbeater.cc:503] Master 127.0.97.62:43125 was elected leader, sending a full tablet report...
I20260619 01:00:03.275533 5509 raft_consensus.cc:504] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.276072 5506 ts_tablet_manager.cc:1428] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260619 01:00:03.278985 5509 leader_election.cc:288] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f15f702a75094995abce1a4f505d9673 (127.0.97.3:38841), 83962273cff243109eb68f737c612a64 (127.0.97.2:40229)
I20260619 01:00:03.281415 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.278507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56023 (local address 127.0.97.2:40229)
0619 01:00:03.278677 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.278683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.278990 (+ 307us) server_negotiation.cc:407] Connection header received
0619 01:00:03.279272 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.279280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.279346 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.279410 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.280704 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.280711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.280719 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.280740 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.280775 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.280791 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.280894 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.281088 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.281151 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:03.281533 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.278029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40229 (local address 127.0.97.1:56023)
0619 01:00:03.278860 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.278892 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.279096 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.279626 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.279632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.279667 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.280519 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.280524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.280540 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.280586 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.280941 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.280946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.280951 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.281104 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.281145 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":556,"threads_started":1}
I20260619 01:00:03.282485 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.279744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60913 (local address 127.0.97.3:38841)
0619 01:00:03.279900 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.279904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.280486 (+ 582us) server_negotiation.cc:407] Connection header received
0619 01:00:03.280620 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.280628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.280687 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.280773 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.281882 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.281887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.281890 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.281919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.281965 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.281975 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.282098 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.282193 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.282251 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:00:03.282516 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.279792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38841 (local address 127.0.97.1:60913)
0619 01:00:03.280368 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.280386 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:03.280493 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:03.280892 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:03.280896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:03.280917 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:03.281679 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:03.281685 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:03.281702 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:03.281748 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:03.282074 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:03.282079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:03.282084 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:03.282206 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 01:00:03.282241 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":326,"threads_started":1}
I20260619 01:00:03.282533 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c7e01831034fb3b1319bc2fe4dce25" candidate_uuid: "bdc252ce63da471d90cdaa58eb1b3631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83962273cff243109eb68f737c612a64" is_pre_election: true
I20260619 01:00:03.283114 5309 raft_consensus.cc:2454] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc252ce63da471d90cdaa58eb1b3631 in term 0.
I20260619 01:00:03.283623 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c7e01831034fb3b1319bc2fe4dce25" candidate_uuid: "bdc252ce63da471d90cdaa58eb1b3631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15f702a75094995abce1a4f505d9673" is_pre_election: true
I20260619 01:00:03.283741 5113 leader_election.cc:302] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83962273cff243109eb68f737c612a64, bdc252ce63da471d90cdaa58eb1b3631; no voters:
I20260619 01:00:03.284087 5439 raft_consensus.cc:2454] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bdc252ce63da471d90cdaa58eb1b3631 in term 0.
I20260619 01:00:03.284101 5510 raft_consensus.cc:2781] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:03.284257 5510 raft_consensus.cc:482] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:03.284384 5510 raft_consensus.cc:3037] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:03.286777 5510 raft_consensus.cc:504] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
I20260619 01:00:03.287438 5510 leader_election.cc:288] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [CANDIDATE]: Term 1 election: Requested vote from peers f15f702a75094995abce1a4f505d9673 (127.0.97.3:38841), 83962273cff243109eb68f737c612a64 (127.0.97.2:40229)
I20260619 01:00:03.287801 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c7e01831034fb3b1319bc2fe4dce25" candidate_uuid: "bdc252ce63da471d90cdaa58eb1b3631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15f702a75094995abce1a4f505d9673"
I20260619 01:00:03.287886 5309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c7e01831034fb3b1319bc2fe4dce25" candidate_uuid: "bdc252ce63da471d90cdaa58eb1b3631" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83962273cff243109eb68f737c612a64"
I20260619 01:00:03.288035 5439 raft_consensus.cc:3037] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:03.288132 5309 raft_consensus.cc:3037] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:03.290526 5439 raft_consensus.cc:2454] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc252ce63da471d90cdaa58eb1b3631 in term 1.
I20260619 01:00:03.290521 5309 raft_consensus.cc:2454] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bdc252ce63da471d90cdaa58eb1b3631 in term 1.
I20260619 01:00:03.290930 5113 leader_election.cc:302] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83962273cff243109eb68f737c612a64, bdc252ce63da471d90cdaa58eb1b3631; no voters:
I20260619 01:00:03.291249 5510 raft_consensus.cc:2781] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:03.291455 5510 raft_consensus.cc:686] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 1 LEADER]: Becoming Leader. State: Replica: bdc252ce63da471d90cdaa58eb1b3631, State: Running, Role: LEADER
I20260619 01:00:03.291946 5510 consensus_queue.cc:237] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } }
W20260619 01:00:03.294731 5224 tablet.cc:2369] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:03.297053 5036 catalog_manager.cc:5495] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 reported cstate change: term changed from 0 to 1, leader changed from <none> to bdc252ce63da471d90cdaa58eb1b3631 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "bdc252ce63da471d90cdaa58eb1b3631" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdc252ce63da471d90cdaa58eb1b3631" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260619 01:00:03.324039 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.321895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.0.97.1:46653)
0619 01:00:03.322090 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.322095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.322114 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:03.322337 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.322341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.322427 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.322536 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.323396 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.323402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.323412 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.323428 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.323457 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.323469 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.323586 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.323707 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.323773 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:03.331984 5309 raft_consensus.cc:1261] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 1 FOLLOWER]: Refusing update from remote peer bdc252ce63da471d90cdaa58eb1b3631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:03.331986 5439 raft_consensus.cc:1261] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 1 FOLLOWER]: Refusing update from remote peer bdc252ce63da471d90cdaa58eb1b3631: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:03.332870 5509 consensus_queue.cc:1035] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83962273cff243109eb68f737c612a64" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:03.333165 5510 consensus_queue.cc:1035] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15f702a75094995abce1a4f505d9673" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:03.344515 5516 mvcc.cc:205] Tried to move back new op lower bound from 7298378970438721536 to 7298378970288263168. Current Snapshot: MvccSnapshot[applied={T|T < 7298378970438721536}]
I20260619 01:00:03.357203 5036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:03.357502 5036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:03.361121 5036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b4b1c1a4f924de8b7588a7395f91b16: Sending DeleteTablet for 3 replicas of tablet 11c7e01831034fb3b1319bc2fe4dce25
I20260619 01:00:03.362288 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:03 UTC) from {username='slave'} at 127.0.0.1:37604
I20260619 01:00:03.362447 5289 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:03 UTC) from {username='slave'} at 127.0.0.1:54594
I20260619 01:00:03.362537 5158 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c7e01831034fb3b1319bc2fe4dce25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:03 UTC) from {username='slave'} at 127.0.0.1:45880
I20260619 01:00:03.364565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5096
I20260619 01:00:03.374971 5526 tablet_replica.cc:330] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: stopping tablet replica
I20260619 01:00:03.375487 5526 raft_consensus.cc:2229] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:03.376011 5526 raft_consensus.cc:2258] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:03.376928 5528 tablet_replica.cc:330] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631: stopping tablet replica
I20260619 01:00:03.377882 5527 tablet_replica.cc:330] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: stopping tablet replica
I20260619 01:00:03.378289 5527 raft_consensus.cc:2229] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:03.378863 5527 raft_consensus.cc:2258] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:03.379639 5528 raft_consensus.cc:2229] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:03.379971 5526 ts_tablet_manager.cc:1905] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:03.381011 5527 ts_tablet_manager.cc:1905] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:03.379801 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.366883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45892 (local address 127.0.97.1:46653)
0619 01:00:03.367110 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.367120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.377116 (+ 9996us) server_negotiation.cc:407] Connection header received
0619 01:00:03.377362 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.377367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.377433 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.377541 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.378693 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.378699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.378705 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.378748 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.378794 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.378809 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.378954 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.379408 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.379470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:00:03.383541 5218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:03.385502 5526 ts_tablet_manager.cc:1918] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:03.385771 5526 log.cc:1196] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/11c7e01831034fb3b1319bc2fe4dce25
I20260619 01:00:03.386214 5526 ts_tablet_manager.cc:1939] T 11c7e01831034fb3b1319bc2fe4dce25 P f15f702a75094995abce1a4f505d9673: Deleting consensus metadata
I20260619 01:00:03.387849 5024 catalog_manager.cc:4841] TS f15f702a75094995abce1a4f505d9673 (127.0.97.3:38841): tablet 11c7e01831034fb3b1319bc2fe4dce25 (table test-table [id=d9fc4245fa0348198b84759992791e51]) successfully deleted
I20260619 01:00:03.389144 5527 ts_tablet_manager.cc:1918] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:03.389405 5527 log.cc:1196] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/11c7e01831034fb3b1319bc2fe4dce25
I20260619 01:00:03.389955 5527 ts_tablet_manager.cc:1939] T 11c7e01831034fb3b1319bc2fe4dce25 P 83962273cff243109eb68f737c612a64: Deleting consensus metadata
I20260619 01:00:03.391498 5023 catalog_manager.cc:4841] TS 83962273cff243109eb68f737c612a64 (127.0.97.2:40229): tablet 11c7e01831034fb3b1319bc2fe4dce25 (table test-table [id=d9fc4245fa0348198b84759992791e51]) successfully deleted
I20260619 01:00:03.478221 5528 raft_consensus.cc:2258] T 11c7e01831034fb3b1319bc2fe4dce25 P bdc252ce63da471d90cdaa58eb1b3631 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:03.478792 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5096
I20260619 01:00:03.496390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5227
W20260619 01:00:03.496343 5024 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46653 (error 108)
I20260619 01:00:03.498204 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.497726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45906)
0619 01:00:03.497910 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.498045 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 01:00:03.498448 5024 catalog_manager.cc:4584] TS bdc252ce63da471d90cdaa58eb1b3631 (127.0.97.1:46653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11c7e01831034fb3b1319bc2fe4dce25: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
I20260619 01:00:03.499357 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.497425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.0.97.2:40229)
0619 01:00:03.497618 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.497625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.497644 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:03.497705 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.497708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.497819 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.497924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.498806 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.498812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.498817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.498834 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.498876 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.498889 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.498992 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.499113 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.499156 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:00:03.499949 5349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:03.552342 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.551797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45922)
0619 01:00:03.552057 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.552165 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 01:00:03.553416 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.553007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45924)
0619 01:00:03.553158 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.553252 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 01:00:03.589082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5227
I20260619 01:00:03.605664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5357
I20260619 01:00:03.608268 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.606424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.0.97.3:38841)
0619 01:00:03.606601 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:03.606605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:03.606623 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:00:03.606779 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:03.606788 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:03.606876 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:03.606967 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:03.607751 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:03.607755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:03.607758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:03.607773 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:03.607803 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:03.607815 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:03.607917 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:03.607980 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 01:00:03.608051 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:00:03.608896 5479 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:03.613735 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.613189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45926)
0619 01:00:03.613436 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.613551 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 01:00:03.614687 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.614365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45936)
0619 01:00:03.614503 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.614574 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 01:00:03.695382 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.694878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45940)
0619 01:00:03.695084 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.695190 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 01:00:03.695823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5357
I20260619 01:00:03.696655 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:03.696094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46653 (local address 127.0.0.1:45942)
0619 01:00:03.696351 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:03.696442 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260619 01:00:03.712435 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5006
I20260619 01:00:03.713532 5067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:03.817962 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5006
2026-06-19T01:00:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2506 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 01:00:03.858196 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:00:03.860707 388 webserver.cc:460] Webserver started at https://127.0.0.1:37357/ using document root <none> and password file <none>
I20260619 01:00:03.861033 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:00:03.861660 388 webserver.cc:460] Webserver started at http://127.0.0.1:46289/ using document root <none> and password file <none>
2026-06-19T01:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:03Z Disabled control of system clock
I20260619 01:00:03.887399 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42453
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/testchainca.cert with env {}
W20260619 01:00:04.198608 5540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:04.198952 5540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:04.199048 5540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:04.206060 5540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:04.206151 5540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:04.206188 5540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:04.206243 5540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:04.206285 5540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:00:04.206344 5540 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:00:04.206382 5540 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37357/jwks.json
I20260619 01:00:04.215963 5540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:04.217116 5540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:04.218350 5540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:04.229847 5548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:04.229947 5545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:04.229979 5546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:04.230922 5540 server_base.cc:1034] running on GCE node
I20260619 01:00:04.231777 5540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:04.233220 5540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:04.234438 5540 hybrid_clock.cc:648] HybridClock initialized: now 1781830804234398 us; error 56 us; skew 500 ppm
I20260619 01:00:04.234862 5540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:04.237015 5540 webserver.cc:460] Webserver started at http://127.0.97.62:40323/ using document root <none> and password file <none>
I20260619 01:00:04.237709 5540 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:04.237833 5540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:04.238138 5540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:04.240976 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "290354c6b27b43908f111b7d87832e16"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "1f8a7cff88a7ea497cc24e2dcfea9360"
server_key_iv: "423669cc25ee5ae73b975809c8836331"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.241724 5540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "290354c6b27b43908f111b7d87832e16"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "1f8a7cff88a7ea497cc24e2dcfea9360"
server_key_iv: "423669cc25ee5ae73b975809c8836331"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.246596 5540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 01:00:04.249835 5554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:04.251258 5540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:04.251433 5540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "290354c6b27b43908f111b7d87832e16"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "1f8a7cff88a7ea497cc24e2dcfea9360"
server_key_iv: "423669cc25ee5ae73b975809c8836331"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.251641 5540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:04.261678 5540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:04.262382 5540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:04.262645 5540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:04.280203 5540 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42453
I20260619 01:00:04.280210 5605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42453 every 8 connection(s)
I20260619 01:00:04.281452 5540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:04.284179 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5540
I20260619 01:00:04.284387 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:04.284981 388 external_mini_cluster.cc:1428] Setting key 35a056d5a28dc06356e86407e5c0b94a
I20260619 01:00:04.285198 5606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:04.299825 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:04.286822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56434 (local address 127.0.97.62:42453)
0619 01:00:04.287924 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:04.287938 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:04.287989 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:00:04.290013 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:04.290039 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:04.290628 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:04.291054 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:04.291583 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.293490 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.294895 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.295923 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.296131 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:04.297194 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:04.297204 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:04.297221 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:04.297940 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:04.298031 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:04.298055 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:04.298163 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:04.298311 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 01:00:04.298607 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":356,"threads_started":1}
I20260619 01:00:04.300004 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16: Bootstrap starting.
I20260619 01:00:04.303253 5606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:04.303344 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:00:04.304210 5606 log.cc:824] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:04.307049 5606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16: No bootstrap required, opened a new log
I20260619 01:00:04.314077 5606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } }
I20260619 01:00:04.314376 5606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:04.314459 5606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290354c6b27b43908f111b7d87832e16, State: Initialized, Role: FOLLOWER
I20260619 01:00:04.314993 5606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } }
I20260619 01:00:04.315234 5606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:04.315371 5606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:04.315562 5606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:04.317660 5606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } }
I20260619 01:00:04.318094 5606 leader_election.cc:302] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 290354c6b27b43908f111b7d87832e16; no voters:
I20260619 01:00:04.318569 5606 leader_election.cc:288] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:04.318753 5611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:04.319397 5611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [term 1 LEADER]: Becoming Leader. State: Replica: 290354c6b27b43908f111b7d87832e16, State: Running, Role: LEADER
I20260619 01:00:04.319855 5611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } }
I20260619 01:00:04.320550 5606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:04.322877 5613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 290354c6b27b43908f111b7d87832e16. Latest consensus state: current_term: 1 leader_uuid: "290354c6b27b43908f111b7d87832e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } } }
I20260619 01:00:04.323307 5613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:04.323547 5612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "290354c6b27b43908f111b7d87832e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290354c6b27b43908f111b7d87832e16" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42453 } } }
I20260619 01:00:04.323830 5612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:04.325841 5620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:04.329648 5620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:04.337001 5620 catalog_manager.cc:1347] Generated new cluster ID: 27d5213bf26142e995b6c308894d604d
I20260619 01:00:04.337105 5620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:04.377761 5620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:04.379143 5620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:04.389102 5620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 290354c6b27b43908f111b7d87832e16: Generated new TSK 0
I20260619 01:00:04.389889 5620 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 01:00:04.432165 5636 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 01:00:04.432330 5636 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:00:04.433233 5609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:04.412201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.0.97.62:42453)
0619 01:00:04.412429 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:04.412438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:04.412986 (+ 548us) server_negotiation.cc:407] Connection header received
0619 01:00:04.413269 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:04.413274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:04.413341 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:04.413457 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:00:04.414072 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.414826 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.416963 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.417396 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.417490 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:04.418065 (+ 575us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:00:04.432750 (+ 14685us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 01:00:04.433002 (+ 252us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56448: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:00:04.433373 5609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56448: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 01:00:04.435261 5635 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:04.412185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42453 (local address 127.0.0.1:56448)
0619 01:00:04.412876 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:04.412899 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:04.413073 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:04.413512 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:04.413526 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:04.413538 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:00:04.413928 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:04.413941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.414961 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.414967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:04.416736 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:04.416750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:04.417512 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:04.417581 (+ 69us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:04.417758 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:04.417832 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:00:04.433014 (+ 15182us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 01:00:04.433351 (+ 337us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":358,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:42453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 01:00:04.436977 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5540
I20260619 01:00:04.438014 5601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:04.530108 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5540
2026-06-19T01:00:04Z chronyd exiting
I20260619 01:00:04.579599 388 test_util.cc:175] -----------------------------------------------
I20260619 01:00:04.579732 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781830760188020-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (725 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 01:00:04.583345 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:00:04.585017 388 webserver.cc:460] Webserver started at https://127.0.0.1:34183/ using document root <none> and password file <none>
I20260619 01:00:04.585229 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:00:04.585842 388 webserver.cc:460] Webserver started at http://127.0.0.1:39919/ using document root <none> and password file <none>
2026-06-19T01:00:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:04Z Disabled control of system clock
I20260619 01:00:04.610236 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39913
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39913
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34183/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/testchainca.cert with env {}
W20260619 01:00:04.920192 5643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:04.920476 5643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:04.920563 5643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:04.927615 5643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:04.927728 5643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:04.927767 5643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:04.927829 5643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:04.927875 5643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:00:04.927930 5643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:00:04.927963 5643 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34183/jwks.json
I20260619 01:00:04.937678 5643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39913
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34183/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:04.938910 5643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:04.940173 5643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:04.952179 5651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:04.952262 5649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:04.952216 5648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:04.960314 5643 server_base.cc:1034] running on GCE node
I20260619 01:00:04.961146 5643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:04.962601 5643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:04.963829 5643 hybrid_clock.cc:648] HybridClock initialized: now 1781830804963798 us; error 46 us; skew 500 ppm
I20260619 01:00:04.964177 5643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:04.966526 5643 webserver.cc:460] Webserver started at http://127.0.97.62:33233/ using document root <none> and password file <none>
I20260619 01:00:04.967329 5643 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:04.967463 5643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:04.967782 5643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:04.970583 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "e3d7300f72b69ab43126c0314a1aa772"
server_key_iv: "ee850d5005c11c0f97d07873b46078b6"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.971477 5643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "e3d7300f72b69ab43126c0314a1aa772"
server_key_iv: "ee850d5005c11c0f97d07873b46078b6"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.976506 5643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:00:04.979846 5657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:04.981103 5643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:04.981274 5643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4"
format_stamp: "Formatted at 2026-06-19 01:00:04 on dist-test-slave-3rxd"
server_key: "e3d7300f72b69ab43126c0314a1aa772"
server_key_iv: "ee850d5005c11c0f97d07873b46078b6"
server_key_version: "encryptionkey@0"
I20260619 01:00:04.981467 5643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:04.997413 5643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:04.998144 5643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:04.998409 5643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:05.015650 5643 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39913
I20260619 01:00:05.015681 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39913 every 8 connection(s)
I20260619 01:00:05.018357 5643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:05.021858 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:05.027181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5643
I20260619 01:00:05.027370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:05.028208 388 external_mini_cluster.cc:1428] Setting key c9fd1a25589cb09e1b0cea1b60308d58
I20260619 01:00:05.032994 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4: Bootstrap starting.
I20260619 01:00:05.035992 5709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:05.036963 5709 log.cc:824] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:05.039731 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4: No bootstrap required, opened a new log
I20260619 01:00:05.041044 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:05.030110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.0.97.62:39913)
0619 01:00:05.031203 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:05.031213 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:05.031255 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:00:05.032833 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:05.032854 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:05.033352 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:05.033718 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:05.034246 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.035553 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.036855 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.037562 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.037785 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.038924 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:05.038932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:05.038945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:05.039451 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:05.039527 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:05.039547 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:05.039679 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:05.039876 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 01:00:05.040173 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":393,"threads_started":1}
I20260619 01:00:05.044595 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:00:05.047538 5709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } }
I20260619 01:00:05.047932 5709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:05.048049 5709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b090a988c1f4bafaf2d9e94c5ff14b4, State: Initialized, Role: FOLLOWER
I20260619 01:00:05.048619 5709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } }
I20260619 01:00:05.048866 5709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:05.049013 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:05.049192 5709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:05.051654 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } }
I20260619 01:00:05.052122 5709 leader_election.cc:302] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b090a988c1f4bafaf2d9e94c5ff14b4; no voters:
I20260619 01:00:05.052650 5709 leader_election.cc:288] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:05.052788 5714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:05.053534 5714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b090a988c1f4bafaf2d9e94c5ff14b4, State: Running, Role: LEADER
I20260619 01:00:05.054078 5714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } }
I20260619 01:00:05.054826 5709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:05.056797 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b090a988c1f4bafaf2d9e94c5ff14b4. Latest consensus state: current_term: 1 leader_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } } }
I20260619 01:00:05.056821 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b090a988c1f4bafaf2d9e94c5ff14b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39913 } } }
I20260619 01:00:05.057253 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:05.057253 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:05.059468 5719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:05.064522 5719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:05.073442 5719 catalog_manager.cc:1347] Generated new cluster ID: 77f5f8fcda85404aba16e7ad9616b3df
I20260619 01:00:05.073556 5719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:05.096951 5719 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:05.098426 5719 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:05.108687 5719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b090a988c1f4bafaf2d9e94c5ff14b4: Generated new TSK 0
I20260619 01:00:05.109578 5719 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 01:00:05.150791 5739 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 01:00:05.150913 5739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:00:05.152108 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:05.132087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45502 (local address 127.0.97.62:39913)
0619 01:00:05.132273 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:05.132279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:05.132647 (+ 368us) server_negotiation.cc:407] Connection header received
0619 01:00:05.132837 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:05.132843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:05.132930 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:05.133106 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:00:05.133472 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.134606 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.135993 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.136677 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.136803 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.137237 (+ 434us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:00:05.151465 (+ 14228us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 01:00:05.151823 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 01:00:05.152158 5738 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:05.131960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39913 (local address 127.0.0.1:45502)
0619 01:00:05.132549 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:05.132565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:05.132675 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:05.133109 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:05.133112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:05.133120 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:00:05.133344 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:05.133350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.134763 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.134768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:05.135853 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:05.135863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.136791 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.136828 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:05.136952 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.136981 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:00:05.151676 (+ 14695us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 01:00:05.151855 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:39913: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":336,"threads_started":1}
W20260619 01:00:05.152227 5712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45502: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34183/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260619 01:00:05.152845 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5643
I20260619 01:00:05.153910 5704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:05.246256 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5643
2026-06-19T01:00:05Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (719 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 01:00:05.302773 388 mini_oidc.cc:152] Starting JWKS server
I20260619 01:00:05.304605 388 webserver.cc:460] Webserver started at https://127.0.0.1:41931/ using document root <none> and password file <none>
I20260619 01:00:05.304842 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 01:00:05.305408 388 webserver.cc:460] Webserver started at http://127.0.0.1:38455/ using document root <none> and password file <none>
2026-06-19T01:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:05Z Disabled control of system clock
I20260619 01:00:05.330403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37789
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37789
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41931/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 01:00:05.645961 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:05.646273 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:05.646430 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:05.654088 5746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:05.654225 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:05.654353 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:05.654466 5746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:05.654533 5746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:00:05.654618 5746 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:00:05.654696 5746 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41931/jwks.json
I20260619 01:00:05.664038 5746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37789
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41931/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:05.665380 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:05.667238 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:05.682904 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:05.683066 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:05.682920 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:05.683681 5746 server_base.cc:1034] running on GCE node
I20260619 01:00:05.684512 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:05.685951 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:05.687152 5746 hybrid_clock.cc:648] HybridClock initialized: now 1781830805687118 us; error 42 us; skew 500 ppm
I20260619 01:00:05.687500 5746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:05.695868 5746 webserver.cc:460] Webserver started at http://127.0.97.62:41065/ using document root <none> and password file <none>
I20260619 01:00:05.696655 5746 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:05.696776 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:05.697134 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:05.700278 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "6aaa5703fc85492c8a488a9613932700"
format_stamp: "Formatted at 2026-06-19 01:00:05 on dist-test-slave-3rxd"
server_key: "53be618cda1fc8403f88d1c01abfe6a0"
server_key_iv: "04451239a1db10ff8ee9a989db7c3ac8"
server_key_version: "encryptionkey@0"
I20260619 01:00:05.701100 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "6aaa5703fc85492c8a488a9613932700"
format_stamp: "Formatted at 2026-06-19 01:00:05 on dist-test-slave-3rxd"
server_key: "53be618cda1fc8403f88d1c01abfe6a0"
server_key_iv: "04451239a1db10ff8ee9a989db7c3ac8"
server_key_version: "encryptionkey@0"
I20260619 01:00:05.708176 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 01:00:05.713128 5760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:05.714926 5746 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:00:05.715127 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "6aaa5703fc85492c8a488a9613932700"
format_stamp: "Formatted at 2026-06-19 01:00:05 on dist-test-slave-3rxd"
server_key: "53be618cda1fc8403f88d1c01abfe6a0"
server_key_iv: "04451239a1db10ff8ee9a989db7c3ac8"
server_key_version: "encryptionkey@0"
I20260619 01:00:05.715385 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:05.737573 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:05.738394 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:05.738646 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:05.756821 5746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37789
I20260619 01:00:05.756824 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37789 every 8 connection(s)
I20260619 01:00:05.758210 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:05.762234 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:05.766386 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5746
I20260619 01:00:05.766601 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:05.767284 388 external_mini_cluster.cc:1428] Setting key 79944ba6f035e26a15a2fbea3095cc8a
I20260619 01:00:05.773389 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700: Bootstrap starting.
I20260619 01:00:05.776928 5812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:05.778415 5812 log.cc:824] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:05.781628 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:05.769342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.0.97.62:37789)
0619 01:00:05.770355 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:05.770365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:05.770406 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:00:05.772097 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:05.772120 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:05.772574 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:05.772884 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:05.773459 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.774927 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.776681 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.777436 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.777577 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.778913 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:05.778923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:05.778941 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:05.779787 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:05.779891 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:05.779928 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:05.780024 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:05.780320 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 01:00:05.780673 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260619 01:00:05.781670 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700: No bootstrap required, opened a new log
I20260619 01:00:05.785733 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:00:05.790531 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } }
I20260619 01:00:05.790995 5812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:05.791114 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaa5703fc85492c8a488a9613932700, State: Initialized, Role: FOLLOWER
I20260619 01:00:05.791805 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } }
I20260619 01:00:05.792098 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:05.792255 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:05.792433 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:05.794611 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } }
I20260619 01:00:05.795128 5812 leader_election.cc:302] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aaa5703fc85492c8a488a9613932700; no voters:
I20260619 01:00:05.795670 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:05.795828 5817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:05.796504 5817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [term 1 LEADER]: Becoming Leader. State: Replica: 6aaa5703fc85492c8a488a9613932700, State: Running, Role: LEADER
I20260619 01:00:05.797122 5817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } }
I20260619 01:00:05.797974 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:05.800088 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aaa5703fc85492c8a488a9613932700. Latest consensus state: current_term: 1 leader_uuid: "6aaa5703fc85492c8a488a9613932700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } } }
I20260619 01:00:05.800088 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aaa5703fc85492c8a488a9613932700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aaa5703fc85492c8a488a9613932700" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37789 } } }
I20260619 01:00:05.800513 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:05.800568 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:05.803395 5824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:05.808526 5824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:05.817286 5824 catalog_manager.cc:1347] Generated new cluster ID: d9d365553b334330a3b6b6ecc71a13dd
I20260619 01:00:05.817394 5824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:05.849877 5824 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:05.851329 5824 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:05.861879 5824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6aaa5703fc85492c8a488a9613932700: Generated new TSK 0
I20260619 01:00:05.862752 5824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:05.891901 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:05.883346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.0.97.62:37789)
0619 01:00:05.883556 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:05.883566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:05.884083 (+ 517us) server_negotiation.cc:407] Connection header received
0619 01:00:05.884303 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:05.884307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:05.884381 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:05.884506 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:00:05.885017 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.885743 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.887750 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.888216 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.888306 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.888916 (+ 610us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:00:05.891302 (+ 2386us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 01:00:05.891610 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 01:00:05.892028 5815 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55504: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 01:00:05.894078 5841 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:05.883327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37789 (local address 127.0.0.1:55504)
0619 01:00:05.883977 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:05.883996 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:05.884141 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:05.884554 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:05.884559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:05.884572 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:00:05.884877 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:05.884890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.885873 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.885879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:05.887593 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:05.887609 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:05.888335 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:05.888408 (+ 73us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:05.888598 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:05.888644 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:00:05.891515 (+ 2871us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 01:00:05.893619 (+ 2104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37789: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41931/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":1794,"thread_start_us":347,"threads_started":1}
I20260619 01:00:05.895678 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5746
I20260619 01:00:05.896843 5807 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 01:00:05.904050 5842 webserver.cc:533] Webserver: sslize failed:
E20260619 01:00:05.904173 5842 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 01:00:05.987536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5746
2026-06-19T01:00:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T01:00:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:06Z Disabled control of system clock
I20260619 01:00:06.067523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42235
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:36473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42235
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/testchainca.cert with env {}
W20260619 01:00:06.386200 5847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:06.386492 5847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:06.386617 5847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:06.393743 5847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:06.393843 5847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:06.393903 5847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:06.394018 5847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:06.394079 5847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 01:00:06.394181 5847 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 01:00:06.394253 5847 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 01:00:06.404028 5847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:06.405304 5847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:06.406628 5847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:06.418587 5852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:06.418674 5853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:06.418705 5855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:06.419451 5847 server_base.cc:1034] running on GCE node
I20260619 01:00:06.420193 5847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:06.421538 5847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:06.422809 5847 hybrid_clock.cc:648] HybridClock initialized: now 1781830806422773 us; error 48 us; skew 500 ppm
I20260619 01:00:06.423161 5847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:06.425253 5847 webserver.cc:460] Webserver started at http://127.0.97.62:40363/ using document root <none> and password file <none>
I20260619 01:00:06.425954 5847 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:06.426107 5847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:06.426419 5847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:06.429327 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "acc3a2addfa343778df05bf7bd20f397"
format_stamp: "Formatted at 2026-06-19 01:00:06 on dist-test-slave-3rxd"
server_key: "5e6e9a37dc6a290b3e55acdafa87e156"
server_key_iv: "96b3714ec2658185feff8070607b42e0"
server_key_version: "encryptionkey@0"
I20260619 01:00:06.430171 5847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "acc3a2addfa343778df05bf7bd20f397"
format_stamp: "Formatted at 2026-06-19 01:00:06 on dist-test-slave-3rxd"
server_key: "5e6e9a37dc6a290b3e55acdafa87e156"
server_key_iv: "96b3714ec2658185feff8070607b42e0"
server_key_version: "encryptionkey@0"
I20260619 01:00:06.435403 5847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 01:00:06.438702 5861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:06.440140 5847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:00:06.440328 5847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "acc3a2addfa343778df05bf7bd20f397"
format_stamp: "Formatted at 2026-06-19 01:00:06 on dist-test-slave-3rxd"
server_key: "5e6e9a37dc6a290b3e55acdafa87e156"
server_key_iv: "96b3714ec2658185feff8070607b42e0"
server_key_version: "encryptionkey@0"
I20260619 01:00:06.440546 5847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:06.452232 5847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:06.452986 5847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:06.453282 5847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:06.471019 5847 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42235
I20260619 01:00:06.471029 5912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42235 every 8 connection(s)
I20260619 01:00:06.472359 5847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:06.475060 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5847
I20260619 01:00:06.475251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:06.475864 388 external_mini_cluster.cc:1428] Setting key 7444b01df6400321147f86f0d0adcb7c
I20260619 01:00:06.476110 5913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:06.489763 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:06.477859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.0.97.62:42235)
0619 01:00:06.478934 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:06.478945 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:06.478989 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:00:06.480564 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:06.480585 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:06.481038 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:06.481362 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:06.482096 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.483520 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.485378 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.485991 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.486111 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:06.487446 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:06.487455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:06.487468 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:06.488038 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:06.488119 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:06.488138 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:06.488222 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:06.488484 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 01:00:06.488816 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":352,"threads_started":1}
I20260619 01:00:06.490335 5913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397: Bootstrap starting.
I20260619 01:00:06.493577 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:00:06.493933 5913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:06.495023 5913 log.cc:824] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:06.498127 5913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397: No bootstrap required, opened a new log
I20260619 01:00:06.505574 5913 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260619 01:00:06.505954 5913 raft_consensus.cc:374] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:06.506078 5913 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc3a2addfa343778df05bf7bd20f397, State: Initialized, Role: FOLLOWER
I20260619 01:00:06.506757 5913 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260619 01:00:06.507020 5913 raft_consensus.cc:388] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:06.507164 5913 raft_consensus.cc:482] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:06.507331 5913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:06.509559 5913 raft_consensus.cc:504] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260619 01:00:06.510090 5913 leader_election.cc:302] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acc3a2addfa343778df05bf7bd20f397; no voters:
I20260619 01:00:06.510749 5913 leader_election.cc:288] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:06.510912 5918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:06.512048 5918 raft_consensus.cc:686] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [term 1 LEADER]: Becoming Leader. State: Replica: acc3a2addfa343778df05bf7bd20f397, State: Running, Role: LEADER
I20260619 01:00:06.512538 5918 consensus_queue.cc:237] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } }
I20260619 01:00:06.512990 5913 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:06.515419 5919 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acc3a2addfa343778df05bf7bd20f397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } } }
I20260619 01:00:06.515823 5919 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:06.515648 5920 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acc3a2addfa343778df05bf7bd20f397. Latest consensus state: current_term: 1 leader_uuid: "acc3a2addfa343778df05bf7bd20f397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acc3a2addfa343778df05bf7bd20f397" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42235 } } }
I20260619 01:00:06.516004 5920 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:06.516672 5927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:06.522079 5927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:06.531575 5927 catalog_manager.cc:1347] Generated new cluster ID: 5d3306a23a244c9ba5a1303837edd32c
I20260619 01:00:06.531697 5927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:06.558696 5927 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:06.560639 5927 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:06.575030 5927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acc3a2addfa343778df05bf7bd20f397: Generated new TSK 0
I20260619 01:00:06.576056 5927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:06.722608 5916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:06.599532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.0.97.62:42235)
0619 01:00:06.599693 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:06.599717 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:06.600159 (+ 442us) server_negotiation.cc:407] Connection header received
0619 01:00:06.600416 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:06.600420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:06.600491 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:06.600671 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0619 01:00:06.601135 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.601865 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.603340 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.603794 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.603950 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:06.604418 (+ 468us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 01:00:06.722069 (+117651us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 01:00:06.722396 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34932: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 01:00:06.722790 5916 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34932: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 01:00:06.723249 5942 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:06.599514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42235 (local address 127.0.0.1:34932)
0619 01:00:06.600051 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:06.600067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:06.600231 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:06.600698 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:06.600703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:06.600716 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 01:00:06.600979 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:06.600988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.602014 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.602021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:06.603185 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:06.603196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:06.603935 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:06.603989 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:06.604132 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:06.604164 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 01:00:06.722437 (+118273us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 01:00:06.722792 (+ 355us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":292,"threads_started":1}
I20260619 01:00:06.724507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5847
I20260619 01:00:06.725677 5908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:06.827507 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5847
2026-06-19T01:00:06Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (819 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (409 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:07 dist-test-slave-3rxd krb5kdc[5957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:07 dist-test-slave-3rxd krb5kdc[5957](info): set up 2 sockets
Jun 19 01:00:07 dist-test-slave-3rxd krb5kdc[5957](info): commencing operation
krb5kdc: starting...
W20260619 01:00:08.735386 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:08 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:00:08.937568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32795
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:09.253479 5970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:09.253775 5970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:09.253865 5970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:09.260942 5970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:09.261053 5970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:09.261101 5970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:09.261158 5970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:09.261204 5970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:09.270875 5970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:09.272137 5970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:09.273481 5970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:09.285135 5975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:09.285130 5976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:09.285153 5978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:09.286036 5970 server_base.cc:1034] running on GCE node
I20260619 01:00:09.286983 5970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:09.288394 5970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:09.289631 5970 hybrid_clock.cc:648] HybridClock initialized: now 1781830809289594 us; error 44 us; skew 500 ppm
I20260619 01:00:09.289983 5970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:09 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:09.293097 5970 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:00:09.294492 5970 webserver.cc:460] Webserver started at http://127.0.97.62:36127/ using document root <none> and password file <none>
I20260619 01:00:09.295197 5970 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:09.295298 5970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:09.295572 5970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:09.298393 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "cec337156580445c85c9d1cf8ebf7981"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "91bd86e1a3a360d90845bc4546de5467"
server_key_iv: "04e3fe72bc6ccfaf9df4c6834af24459"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.299212 5970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "cec337156580445c85c9d1cf8ebf7981"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "91bd86e1a3a360d90845bc4546de5467"
server_key_iv: "04e3fe72bc6ccfaf9df4c6834af24459"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.304315 5970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:00:09.307811 5985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:09.309159 5970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:09.309299 5970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "cec337156580445c85c9d1cf8ebf7981"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "91bd86e1a3a360d90845bc4546de5467"
server_key_iv: "04e3fe72bc6ccfaf9df4c6834af24459"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.309459 5970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:09.323297 5970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:09.327659 5970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:09.327886 5970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:09.345929 5970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32795
I20260619 01:00:09.345952 6036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32795 every 8 connection(s)
I20260619 01:00:09.347471 5970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:09.351291 6037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:09.356587 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 5970
I20260619 01:00:09.356783 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:09.357398 388 external_mini_cluster.cc:1428] Setting key bb97accb89894af3226f966f6cf47e4d
I20260619 01:00:09.362293 6037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981: Bootstrap starting.
I20260619 01:00:09.366137 6037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:09.367270 6037 log.cc:824] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:09.370281 6037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981: No bootstrap required, opened a new log
Jun 19 01:00:09 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:09.381337 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:09.359539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.97.62:32795)
0619 01:00:09.360397 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:09.360408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:09.360451 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:00:09.362134 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:09.362157 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:09.362626 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:09.363021 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:09.364831 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.366120 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.367506 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.368410 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.368630 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:09.372694 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:09.372704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:09.372730 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:09.372780 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:09.376868 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:09.377533 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:09.377547 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:09.377556 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:09.377764 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:09.378129 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:09.378134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:09.378137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:09.378897 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:09.379213 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:09.379514 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 01:00:09.379968 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":420,"threads_started":1}
I20260619 01:00:09.381089 6037 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } }
I20260619 01:00:09.381528 6037 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:09.381642 6037 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec337156580445c85c9d1cf8ebf7981, State: Initialized, Role: FOLLOWER
I20260619 01:00:09.382273 6037 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } }
I20260619 01:00:09.382543 6037 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:09.382797 6037 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:09.383020 6037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:09.385560 6037 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } }
I20260619 01:00:09.386044 6037 leader_election.cc:302] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec337156580445c85c9d1cf8ebf7981; no voters:
I20260619 01:00:09.386611 6037 leader_election.cc:288] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:09.386803 6042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:09.388180 6042 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [term 1 LEADER]: Becoming Leader. State: Replica: cec337156580445c85c9d1cf8ebf7981, State: Running, Role: LEADER
I20260619 01:00:09.388638 6042 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } }
I20260619 01:00:09.388880 6037 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:09.391422 6043 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec337156580445c85c9d1cf8ebf7981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } } }
I20260619 01:00:09.391599 6044 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec337156580445c85c9d1cf8ebf7981. Latest consensus state: current_term: 1 leader_uuid: "cec337156580445c85c9d1cf8ebf7981" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec337156580445c85c9d1cf8ebf7981" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32795 } } }
I20260619 01:00:09.391819 6043 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:09.391835 6044 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:09.395057 6050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:09.400429 6050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:09.409276 6050 catalog_manager.cc:1347] Generated new cluster ID: 3982ae735ba34d6f9374ccbeaec38f33
I20260619 01:00:09.409390 6050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:09.449599 6050 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:09.451022 6050 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:09.461933 6050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec337156580445c85c9d1cf8ebf7981: Generated new TSK 0
I20260619 01:00:09.462842 6050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:00:09.526742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:09.848644 6063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:09.848914 6063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:09.849084 6063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:09.856118 6063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:09.856221 6063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:09.856428 6063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:09.865780 6063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:09.867120 6063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:09.868420 6063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:09.882140 6069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:09.882372 6068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:09.882650 6071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:09.883419 6063 server_base.cc:1034] running on GCE node
I20260619 01:00:09.884054 6063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:09.884959 6063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:09.886183 6063 hybrid_clock.cc:648] HybridClock initialized: now 1781830809886133 us; error 45 us; skew 500 ppm
I20260619 01:00:09.886498 6063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:09 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:09.889873 6063 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:00:09.891381 6063 webserver.cc:460] Webserver started at http://127.0.97.1:40223/ using document root <none> and password file <none>
I20260619 01:00:09.892048 6063 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:09.892184 6063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:09.892479 6063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:09.895500 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "e889aacfb4504b5b86a3349d079fb203"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "b6f62d09e857acfdfa4f25d008875542"
server_key_iv: "07c27b5baa83a418c2762808b23c53f8"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.896248 6063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e889aacfb4504b5b86a3349d079fb203"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "b6f62d09e857acfdfa4f25d008875542"
server_key_iv: "07c27b5baa83a418c2762808b23c53f8"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.901145 6063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:09.904438 6078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:09.905715 6063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:00:09.905880 6063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "e889aacfb4504b5b86a3349d079fb203"
format_stamp: "Formatted at 2026-06-19 01:00:09 on dist-test-slave-3rxd"
server_key: "b6f62d09e857acfdfa4f25d008875542"
server_key_iv: "07c27b5baa83a418c2762808b23c53f8"
server_key_version: "encryptionkey@0"
I20260619 01:00:09.906087 6063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:09.923216 6063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:09.927052 6063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:09.927320 6063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:09.928201 6063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:09.929477 6063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:09.929575 6063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:09.929740 6063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:09.929811 6063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:09.961921 6063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40923
I20260619 01:00:09.961946 6190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40923 every 8 connection(s)
I20260619 01:00:09.963416 6063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:09.969977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6063
I20260619 01:00:09.970184 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:09.970872 388 external_mini_cluster.cc:1428] Setting key 9cdc0723c27d86d7d0650ffa22ad7f68
Jun 19 01:00:09 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:09.984151 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:09.965948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36685 (local address 127.0.97.62:32795)
0619 01:00:09.966379 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:09.966384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:09.967275 (+ 891us) server_negotiation.cc:407] Connection header received
0619 01:00:09.969035 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:09.969041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:09.969131 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:09.969255 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:09.970912 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.971786 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.973493 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.973990 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.974119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:09.978459 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:09.978465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:09.978468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:09.978498 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:09.981083 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:09.981833 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:09.981839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:09.981842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:09.981967 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:09.982393 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:09.982398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:09.982401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:09.982656 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:09.982844 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:09.983561 (+ 717us) server_negotiation.cc:299] Negotiation successful
0619 01:00:09.983784 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":250,"threads_started":1}
I20260619 01:00:09.985280 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:09.966365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32795 (local address 127.0.97.1:36685)
0619 01:00:09.967119 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:09.967157 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:09.968764 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:09.969498 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:09.969528 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:09.970043 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:09.970699 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:09.970769 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.971967 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.971972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:09.973342 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:09.973354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:09.974220 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:09.974227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:09.974455 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:09.975186 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:09.975222 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:09.978230 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:09.981240 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:09.981250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:09.981268 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:09.981703 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:09.982077 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:09.982083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:09.982087 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:09.982274 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:09.982837 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:09.982845 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:09.983190 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 01:00:09.983470 (+ 280us) client_negotiation.cc:240] Negotiation successful
0619 01:00:09.983990 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":333,"threads_started":1}
I20260619 01:00:09.987046 6191 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32795
I20260619 01:00:09.987351 6191 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:09.988155 6191 heartbeater.cc:511] Master 127.0.97.62:32795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:00:09.990547 6002 ts_manager.cc:194] Registered new tserver with Master: e889aacfb4504b5b86a3349d079fb203 (127.0.97.1:40923)
I20260619 01:00:09.991995 6002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:36685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:00:10.017668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:10.336190 6197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:10.336514 6197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:10.336687 6197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:10.343832 6197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:10.343993 6197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:10.344202 6197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:10.353806 6197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:10.355078 6197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:10.356366 6197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:10.369427 6202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:10.369419 6203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:10.369419 6205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:10.370160 6197 server_base.cc:1034] running on GCE node
I20260619 01:00:10.370826 6197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:10.371687 6197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:10.372879 6197 hybrid_clock.cc:648] HybridClock initialized: now 1781830810372843 us; error 26 us; skew 500 ppm
I20260619 01:00:10.373191 6197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:10 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:10.376259 6197 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:00:10.377556 6197 webserver.cc:460] Webserver started at http://127.0.97.2:44261/ using document root <none> and password file <none>
I20260619 01:00:10.378216 6197 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:10.378355 6197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:10.378633 6197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:10.381418 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "7826e848c51d4edc8fae3470a7c7734c"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "41f64700d7b933f773aa808e2b3782e1"
server_key_iv: "ae8d0074fe96195f3d08b3fd1ce64277"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.382213 6197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7826e848c51d4edc8fae3470a7c7734c"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "41f64700d7b933f773aa808e2b3782e1"
server_key_iv: "ae8d0074fe96195f3d08b3fd1ce64277"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.387209 6197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 01:00:10.390581 6212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.391836 6197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:10.391997 6197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "7826e848c51d4edc8fae3470a7c7734c"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "41f64700d7b933f773aa808e2b3782e1"
server_key_iv: "ae8d0074fe96195f3d08b3fd1ce64277"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.392168 6197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:10.410539 6197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:10.414960 6197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:10.415226 6197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:10.416118 6197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:10.417399 6197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:10.417469 6197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.417644 6197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:10.417696 6197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.448824 6197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33703
I20260619 01:00:10.448871 6324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33703 every 8 connection(s)
I20260619 01:00:10.450095 6197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:10.456662 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6197
I20260619 01:00:10.456848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:10.457286 388 external_mini_cluster.cc:1428] Setting key 6bdc6d2afd9319dd5980aaa4011da8cb
Jun 19 01:00:10 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:10.469570 6193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:10.452670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54717 (local address 127.0.97.62:32795)
0619 01:00:10.452834 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:10.452838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:10.453948 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 01:00:10.455762 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:10.455767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:10.455857 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:10.455964 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:10.457556 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.458423 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.460091 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.460491 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.460642 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:10.464014 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:10.464019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:10.464022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:10.464051 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:10.466642 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:10.467382 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:10.467387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:10.467392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:10.467517 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:10.467944 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:10.467949 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:10.467952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:10.468197 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:10.468339 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:10.469011 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 01:00:10.469229 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:10.470582 6327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:10.453127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32795 (local address 127.0.97.2:54717)
0619 01:00:10.453799 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:10.453840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:10.455492 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:10.456203 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:10.456211 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:10.456691 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:10.457352 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:10.457369 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.458561 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.458567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:10.459939 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:10.459951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.460717 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.460724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:10.460968 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:10.461647 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:10.461673 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:10.463790 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:10.466808 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:10.466817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:10.466834 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:10.467246 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:10.467650 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:10.467655 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:10.467659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:10.467832 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:10.468314 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:10.468322 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:10.468658 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 01:00:10.468934 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 01:00:10.469473 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":274,"threads_started":1}
I20260619 01:00:10.472007 6325 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32795
I20260619 01:00:10.472316 6325 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:10.473050 6325 heartbeater.cc:511] Master 127.0.97.62:32795 requested a full tablet report, sending...
I20260619 01:00:10.474354 6002 ts_manager.cc:194] Registered new tserver with Master: 7826e848c51d4edc8fae3470a7c7734c (127.0.97.2:33703)
I20260619 01:00:10.475180 6002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54717
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:00:10.502923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:10.828719 6330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:10.829021 6330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:10.829113 6330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:10.836637 6330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:10.836753 6330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:10.836867 6330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:10.846727 6330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:32795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:10.848028 6330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:10.849339 6330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:10.862135 6335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:10.862239 6336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:10.862668 6338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:10.863281 6330 server_base.cc:1034] running on GCE node
I20260619 01:00:10.863835 6330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:10.864771 6330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:10.865998 6330 hybrid_clock.cc:648] HybridClock initialized: now 1781830810865922 us; error 66 us; skew 500 ppm
I20260619 01:00:10.866274 6330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:10 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:10.874054 6330 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:00:10.875618 6330 webserver.cc:460] Webserver started at http://127.0.97.3:40237/ using document root <none> and password file <none>
I20260619 01:00:10.876319 6330 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:10.876467 6330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:10.876804 6330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:10.879856 6330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "fc2923811c2d4e7f8ab1a50ab82e69dd"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "19ae32a94d765e31c0ff5c5c3aacf491"
server_key_iv: "89721715adc04fd78a1eb270623e3bbe"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.880638 6330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "fc2923811c2d4e7f8ab1a50ab82e69dd"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "19ae32a94d765e31c0ff5c5c3aacf491"
server_key_iv: "89721715adc04fd78a1eb270623e3bbe"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.885910 6330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:00:10.889338 6345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.890759 6330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:00:10.890935 6330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "fc2923811c2d4e7f8ab1a50ab82e69dd"
format_stamp: "Formatted at 2026-06-19 01:00:10 on dist-test-slave-3rxd"
server_key: "19ae32a94d765e31c0ff5c5c3aacf491"
server_key_iv: "89721715adc04fd78a1eb270623e3bbe"
server_key_version: "encryptionkey@0"
I20260619 01:00:10.891136 6330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:10.930413 6330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:10.934340 6330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:10.934648 6330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:10.935634 6330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:10.936918 6330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:10.936996 6330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.937160 6330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:10.937211 6330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:10.970257 6330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45887
I20260619 01:00:10.970297 6457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45887 every 8 connection(s)
I20260619 01:00:10.971573 6330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:10.973846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6330
I20260619 01:00:10.974028 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:10.974579 388 external_mini_cluster.cc:1428] Setting key 33841883675c741bead576761086debb
Jun 19 01:00:10 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:10.991830 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:10.975623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53385 (local address 127.0.97.62:32795)
0619 01:00:10.976069 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:10.976075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:10.976095 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:00:10.978323 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:10.978326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:10.978386 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:10.978479 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:10.979947 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.980719 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.982138 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.982500 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.982597 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:10.985981 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:10.985986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:10.985989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:10.986018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:10.988879 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:10.989630 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:10.989636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:10.989640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:10.989787 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:10.990317 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:10.990322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:10.990326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:10.990607 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:10.990804 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:10.991419 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 01:00:10.991572 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":567,"threads_started":1}
I20260619 01:00:10.992789 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:10.975034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32795 (local address 127.0.97.3:53385)
0619 01:00:10.975927 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:10.975991 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:10.978064 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:10.978682 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:10.978689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:10.979224 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:10.979768 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:10.979784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.980844 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.980847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:10.982023 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:10.982033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:10.982679 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:10.982684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:10.982968 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:10.983631 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:10.983653 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:10.985747 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:10.989031 (+ 3284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:10.989041 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:10.989057 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:10.989494 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:10.989919 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:10.989923 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:10.989927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:10.990186 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:10.990771 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:10.990779 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:10.991073 (+ 294us) client_negotiation.cc:769] Sending connection context
0619 01:00:10.991286 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 01:00:10.991652 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":389,"threads_started":1}
I20260619 01:00:10.994215 6458 heartbeater.cc:347] Connected to a master server at 127.0.97.62:32795
I20260619 01:00:10.994549 6458 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:10.994962 6191 heartbeater.cc:503] Master 127.0.97.62:32795 was elected leader, sending a full tablet report...
I20260619 01:00:10.995505 6458 heartbeater.cc:511] Master 127.0.97.62:32795 requested a full tablet report, sending...
I20260619 01:00:10.996961 6002 ts_manager.cc:194] Registered new tserver with Master: fc2923811c2d4e7f8ab1a50ab82e69dd (127.0.97.3:45887)
I20260619 01:00:10.997665 6002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53385
I20260619 01:00:11.002357 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/test-admin.keytab.
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.027938 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 01:00:11.032613 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:11.030484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.97.1:40923)
0619 01:00:11.031130 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.031137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.031163 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:00:11.031220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.031226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.031434 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.031571 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.032261 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60510: BlockingRecv error: recv got EOF from 127.0.0.1:60510 (error 108)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":366,"threads_started":1}
W20260619 01:00:11.032823 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.034381 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.035965 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:11.051101 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.037452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.0.97.1:40923)
0619 01:00:11.037687 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.037694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.037954 (+ 260us) server_negotiation.cc:407] Connection header received
0619 01:00:11.038187 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.038192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.038248 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.038347 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.039367 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.040305 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.041551 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.042100 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.042233 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.045772 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.045782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.045796 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.045824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.048444 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.049096 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.049104 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.049106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.049225 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.049588 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.049594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.049598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.049941 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.050231 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.050558 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.050837 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 01:00:11.056597 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:11.054829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.97.1:40923)
0619 01:00:11.055028 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.055040 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.055240 (+ 200us) server_negotiation.cc:407] Connection header received
0619 01:00:11.055420 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.055425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.055498 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.055606 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.056374 (+ 768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60522: BlockingRecv error: recv got EOF from 127.0.0.1:60522 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:00:11.056780 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.058211 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.059444 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:11.072652 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.060619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.97.1:40923)
0619 01:00:11.060814 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.060822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.061084 (+ 262us) server_negotiation.cc:407] Connection header received
0619 01:00:11.061243 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.061248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.061311 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.061400 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.062259 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.062979 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.064176 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.064610 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.064735 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.068002 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.068012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.068015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.068065 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.070274 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.070926 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.070933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.070936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.071062 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.071413 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.071420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.071423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.071675 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.071943 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.072193 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.072381 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 01:00:11.076740 6466 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:11.075030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.97.1:40923)
0619 01:00:11.075230 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.075237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.075499 (+ 262us) server_negotiation.cc:407] Connection header received
0619 01:00:11.075682 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.075687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.075812 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.075901 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.076528 (+ 627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60536: BlockingRecv error: recv got EOF from 127.0.0.1:60536 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 01:00:11.076895 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781830760188020-388-0/krb5kdc/krb5cc' of type FILE
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.078382 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:11.079684 388 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:11.092938 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.080864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.0.97.1:40923)
0619 01:00:11.081054 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.081059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.081219 (+ 160us) server_negotiation.cc:407] Connection header received
0619 01:00:11.081321 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.081324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.081371 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.081444 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.082284 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.083083 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.084207 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.084616 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.084706 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.088004 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.088008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.088012 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.088039 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.090813 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.091374 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.091378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.091380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.091516 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.091811 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.091814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.091816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.092050 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.092269 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.092418 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.092618 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:00:11.094480 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6063
I20260619 01:00:11.106235 6466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.095349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.97.1:40923)
0619 01:00:11.096131 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.096138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.096158 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:00:11.096233 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.096238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.096299 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.096395 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.097646 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.098446 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.099668 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.100079 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.100170 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.101942 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.101947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.101950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.101975 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.104225 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.104768 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.104772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.104774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.104888 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.105193 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.105196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.105198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.105410 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.105580 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.105843 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.105981 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 01:00:11.106863 6186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:11.189586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6063
I20260619 01:00:11.206481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6197
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:00:11.222206 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.207391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.97.2:33703)
0619 01:00:11.207980 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.207989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.208028 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:00:11.208122 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.208138 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.208359 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.208494 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.209551 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.210537 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.211864 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.212410 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.212558 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.216670 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.216683 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.216699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.216735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.219507 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.220184 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.220192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.220194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.220318 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.220723 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.220732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.220734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.221065 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.221372 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.221671 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.221911 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":344,"threads_started":1}
I20260619 01:00:11.223547 6320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:11.305457 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6197
I20260619 01:00:11.320935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6330
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[5957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:00:11.336170 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:11.321979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.0.97.3:45887)
0619 01:00:11.322621 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:11.322628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:11.322656 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:00:11.322778 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:11.322789 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:11.323014 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:11.323149 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:11.324102 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.325075 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.326218 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:11.326905 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:11.326997 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:11.330790 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:11.330821 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:11.330846 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:11.330903 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:11.333673 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.334245 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.334252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.334254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.334365 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:11.334674 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:11.334676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:11.334678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:11.335148 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:11.335365 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:11.335602 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 01:00:11.335847 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":352,"threads_started":1}
I20260619 01:00:11.337402 6453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:11.419449 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6330
I20260619 01:00:11.435197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 5970
I20260619 01:00:11.436357 6032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:11.538420 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 5970
2026-06-19T01:00:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3905 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[6486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[6486](info): set up 2 sockets
Jun 19 01:00:11 dist-test-slave-3rxd krb5kdc[6486](info): commencing operation
krb5kdc: starting...
W20260619 01:00:12.624047 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:12 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 01:00:12.834340 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46621
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:13.149618 6499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:13.149971 6499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:13.150187 6499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:13.157407 6499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:13.157542 6499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:13.157588 6499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:13.157662 6499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:13.157725 6499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:13.167443 6499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:13.168748 6499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:13.170090 6499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:13.181468 6505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:13.181507 6507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:13.181529 6504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:13.182754 6499 server_base.cc:1034] running on GCE node
I20260619 01:00:13.183518 6499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:13.184891 6499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:13.186089 6499 hybrid_clock.cc:648] HybridClock initialized: now 1781830813186037 us; error 59 us; skew 500 ppm
I20260619 01:00:13.186430 6499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:13 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:13.189486 6499 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 01:00:13.190977 6499 webserver.cc:460] Webserver started at http://127.0.97.62:37727/ using document root <none> and password file <none>
I20260619 01:00:13.191656 6499 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:13.191797 6499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:13.192090 6499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:13.195022 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "82345d268ed64aa3957b1563bd6f8d75"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "feb7d519bbc35710e3ee18400cf6dd48"
server_key_iv: "3967066caa7c3eeebc4b7c01512c054d"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.195773 6499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "82345d268ed64aa3957b1563bd6f8d75"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "feb7d519bbc35710e3ee18400cf6dd48"
server_key_iv: "3967066caa7c3eeebc4b7c01512c054d"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.200647 6499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 01:00:13.203893 6514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:13.205067 6499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:13.205240 6499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "82345d268ed64aa3957b1563bd6f8d75"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "feb7d519bbc35710e3ee18400cf6dd48"
server_key_iv: "3967066caa7c3eeebc4b7c01512c054d"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.205425 6499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:13.217725 6499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:13.221522 6499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:13.221977 6499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:13.239660 6499 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46621
I20260619 01:00:13.239691 6565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46621 every 8 connection(s)
I20260619 01:00:13.241020 6499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:13.242304 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6499
I20260619 01:00:13.242503 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:13.243336 388 external_mini_cluster.cc:1428] Setting key d49dff3391e97d3ac9c4326a26dcf762
I20260619 01:00:13.245747 6566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:00:13 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:13.258914 6566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Bootstrap starting.
I20260619 01:00:13.263041 6566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:13.264382 6566 log.cc:824] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:13.266969 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:13.245422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.0.97.62:46621)
0619 01:00:13.246187 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:13.246199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:13.246239 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:00:13.247847 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:13.247867 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:13.248247 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:13.248572 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:13.250334 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.251726 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.253399 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.254034 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.254178 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:13.258701 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:13.258732 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:13.258753 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:13.258796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:13.262559 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:13.263306 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:13.263315 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:13.263325 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:13.263548 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:13.263885 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:13.263890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:13.263893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:13.264536 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:13.264839 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:13.265270 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 01:00:13.265653 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":358,"threads_started":1}
I20260619 01:00:13.267858 6566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: No bootstrap required, opened a new log
I20260619 01:00:13.275460 6566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } }
I20260619 01:00:13.275890 6566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:13.275990 6566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82345d268ed64aa3957b1563bd6f8d75, State: Initialized, Role: FOLLOWER
I20260619 01:00:13.276573 6566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } }
I20260619 01:00:13.276804 6566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:13.276924 6566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:13.277107 6566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:13.279239 6566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } }
I20260619 01:00:13.279714 6566 leader_election.cc:302] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82345d268ed64aa3957b1563bd6f8d75; no voters:
I20260619 01:00:13.280326 6566 leader_election.cc:288] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:13.280483 6571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:13.281190 6571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 LEADER]: Becoming Leader. State: Replica: 82345d268ed64aa3957b1563bd6f8d75, State: Running, Role: LEADER
I20260619 01:00:13.281697 6571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } }
I20260619 01:00:13.282366 6566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:13.284435 6573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82345d268ed64aa3957b1563bd6f8d75. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } }
I20260619 01:00:13.284811 6573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:13.284751 6572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } }
I20260619 01:00:13.285049 6572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:13.286581 6579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:13.291688 6579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:13.300104 6579 catalog_manager.cc:1347] Generated new cluster ID: 288596af42a54b8d8ce48c50288d05f8
I20260619 01:00:13.300203 6579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:13.331009 6579 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:13.332878 6579 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:13.342409 6579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Generated new TSK 0
I20260619 01:00:13.343402 6579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 01:00:13.390687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:13.714305 6592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:13.714596 6592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:13.714807 6592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:13.722033 6592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:13.722155 6592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:13.722348 6592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:13.732338 6592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:13.733644 6592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:13.735002 6592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:13.748119 6597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:13.748054 6600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:13.748095 6598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:13.749001 6592 server_base.cc:1034] running on GCE node
I20260619 01:00:13.749610 6592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:13.750540 6592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:13.751777 6592 hybrid_clock.cc:648] HybridClock initialized: now 1781830813751713 us; error 61 us; skew 500 ppm
I20260619 01:00:13.752094 6592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:13 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:13.755185 6592 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:00:13.756762 6592 webserver.cc:460] Webserver started at http://127.0.97.1:38635/ using document root <none> and password file <none>
I20260619 01:00:13.757653 6592 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:13.757807 6592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:13.758157 6592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:13.761314 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "cb803022fd5945729575948839a387d5"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "f89c5d488f0aa375313a3f2a4c1d5136"
server_key_iv: "cea1b4f70d3365709ebaa3992fefb45a"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.762161 6592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cb803022fd5945729575948839a387d5"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "f89c5d488f0aa375313a3f2a4c1d5136"
server_key_iv: "cea1b4f70d3365709ebaa3992fefb45a"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.767807 6592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260619 01:00:13.771463 6607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:13.772763 6592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:13.772956 6592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "cb803022fd5945729575948839a387d5"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "f89c5d488f0aa375313a3f2a4c1d5136"
server_key_iv: "cea1b4f70d3365709ebaa3992fefb45a"
server_key_version: "encryptionkey@0"
I20260619 01:00:13.773196 6592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:13.807454 6592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:13.812052 6592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:13.812359 6592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:13.813369 6592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:13.814903 6592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:13.815049 6592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:13.815214 6592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:13.815292 6592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:13.849038 6592 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37835
I20260619 01:00:13.849076 6719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37835 every 8 connection(s)
I20260619 01:00:13.850430 6592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:13.859477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6592
I20260619 01:00:13.859709 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:13.860379 388 external_mini_cluster.cc:1428] Setting key d2b67762a520895f1b10150066377b1c
Jun 19 01:00:13 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:13.870328 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:13.852986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41235 (local address 127.0.97.62:46621)
0619 01:00:13.853416 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:13.853420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:13.854317 (+ 897us) server_negotiation.cc:407] Connection header received
0619 01:00:13.856155 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:13.856160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:13.856246 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:13.856359 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:13.857809 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.858534 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.860222 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.860653 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.860790 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:13.864725 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:13.864731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:13.864736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:13.864775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:13.867288 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:13.867994 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:13.868000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:13.868003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:13.868129 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:13.868595 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:13.868601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:13.868604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:13.868860 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:13.869010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:13.869739 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 01:00:13.869951 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":242,"threads_started":1}
I20260619 01:00:13.871392 6723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:13.853515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:41235)
0619 01:00:13.854176 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:13.854213 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:13.855902 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:13.856587 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:13.856596 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:13.857086 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:13.857647 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:13.857664 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.858697 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.858700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:13.860077 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:13.860089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:13.860877 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:13.860885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:13.861117 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:13.861771 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:13.861798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:13.864499 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:13.867439 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:13.867449 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:13.867466 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:13.867870 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:13.868236 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:13.868242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:13.868245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:13.868429 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:13.868996 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:13.869004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:13.869340 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 01:00:13.869611 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 01:00:13.870144 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":271,"threads_started":1}
I20260619 01:00:13.873076 6720 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:13.873368 6720 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:13.874058 6720 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:13.876133 6531 ts_manager.cc:194] Registered new tserver with Master: cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:13.877679 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41235
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 01:00:13.907929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:14.223091 6726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:14.223352 6726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:14.223443 6726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:14.230544 6726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:14.230646 6726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:14.230774 6726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:14.240186 6726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:14.241359 6726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:14.242604 6726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:14.256433 6731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:14.256409 6732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:14.256611 6734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:14.257025 6726 server_base.cc:1034] running on GCE node
I20260619 01:00:14.257701 6726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:14.258594 6726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:14.259815 6726 hybrid_clock.cc:648] HybridClock initialized: now 1781830814259785 us; error 44 us; skew 500 ppm
I20260619 01:00:14.260226 6726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:14 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830814, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:14.263356 6726 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:00:14.264700 6726 webserver.cc:460] Webserver started at http://127.0.97.2:44091/ using document root <none> and password file <none>
I20260619 01:00:14.265406 6726 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:14.265548 6726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:14.265844 6726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:14.268671 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "46c75eef91c08f6a6c304ef0898e3c8d"
server_key_iv: "49e7e96715b6f008dd72c09024e4f4c2"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.269416 6726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "46c75eef91c08f6a6c304ef0898e3c8d"
server_key_iv: "49e7e96715b6f008dd72c09024e4f4c2"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.274384 6726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:00:14.277876 6741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.279323 6726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:00:14.279501 6726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "46c75eef91c08f6a6c304ef0898e3c8d"
server_key_iv: "49e7e96715b6f008dd72c09024e4f4c2"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.279733 6726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:14.297269 6726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:14.301535 6726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:14.301824 6726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:14.302803 6726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:14.304061 6726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:14.304162 6726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.304313 6726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:14.304394 6726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.336637 6726 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38023
I20260619 01:00:14.336721 6853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38023 every 8 connection(s)
I20260619 01:00:14.337903 6726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:14.346943 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6726
I20260619 01:00:14.347121 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:14.347546 388 external_mini_cluster.cc:1428] Setting key 6ced74c5bbeaa540461a64daa3a416a7
Jun 19 01:00:14 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830814, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:14.359562 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:14.340489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58217 (local address 127.0.97.62:46621)
0619 01:00:14.340662 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:14.340667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:14.341803 (+ 1136us) server_negotiation.cc:407] Connection header received
0619 01:00:14.343761 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:14.343766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:14.343858 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:14.343978 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:14.345434 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.346140 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.347857 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.348341 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.348458 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:14.352450 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:14.352458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:14.352466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:14.352510 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:14.356220 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:14.357228 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:14.357232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:14.357234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:14.357357 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:14.357857 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:14.357860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:14.357862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:14.358111 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:14.358214 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:14.359169 (+ 955us) server_negotiation.cc:299] Negotiation successful
0619 01:00:14.359330 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:00:14.361137 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:14.340970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:58217)
0619 01:00:14.341647 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:14.341689 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:14.343486 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:14.344240 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:14.344248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:14.344728 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:14.345259 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:14.345287 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.346298 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.346302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:14.347725 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:14.347736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.348613 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.348621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:14.348899 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:14.349786 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:14.349812 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:14.352223 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:14.356401 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:14.356412 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:14.356436 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:14.357104 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:14.357461 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:14.357468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:14.357473 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:14.357748 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:14.358229 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:14.358240 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:14.358745 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 01:00:14.359068 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 01:00:14.359652 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":260,"threads_started":1}
I20260619 01:00:14.362651 6854 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:14.362974 6854 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:14.363698 6854 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:14.365135 6531 ts_manager.cc:194] Registered new tserver with Master: b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:14.366000 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58217
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:00:14.397418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:14.709872 6859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:14.710148 6859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:14.710337 6859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:14.717315 6859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:14.717411 6859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:14.717571 6859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:14.727279 6859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46621
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:14.728530 6859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:14.729801 6859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:14.743101 6867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:14.743144 6864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:14.743062 6865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:14.744047 6859 server_base.cc:1034] running on GCE node
I20260619 01:00:14.744695 6859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:14.745664 6859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:14.746881 6859 hybrid_clock.cc:648] HybridClock initialized: now 1781830814746811 us; error 62 us; skew 500 ppm
I20260619 01:00:14.747221 6859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:14 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830814, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:14.750313 6859 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:00:14.751694 6859 webserver.cc:460] Webserver started at http://127.0.97.3:34489/ using document root <none> and password file <none>
I20260619 01:00:14.752339 6859 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:14.752475 6859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:14.752712 6859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:14.755609 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "feebc3f3881547989c6c32faf22115c5"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "22890aa49fa91d8d4e6eb595e5e7e7f4"
server_key_iv: "40e6e09089d6dbee5155a0af4bbc0bb6"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.756299 6859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "feebc3f3881547989c6c32faf22115c5"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "22890aa49fa91d8d4e6eb595e5e7e7f4"
server_key_iv: "40e6e09089d6dbee5155a0af4bbc0bb6"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.761276 6859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:14.764585 6874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.765966 6859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:00:14.766163 6859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "feebc3f3881547989c6c32faf22115c5"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "22890aa49fa91d8d4e6eb595e5e7e7f4"
server_key_iv: "40e6e09089d6dbee5155a0af4bbc0bb6"
server_key_version: "encryptionkey@0"
I20260619 01:00:14.766389 6859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:14.777544 6859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:14.781476 6859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:14.781777 6859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:14.782801 6859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:14.784310 6859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:14.784379 6859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.784458 6859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:14.784523 6859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:14.817500 6859 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46017
I20260619 01:00:14.817555 6986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46017 every 8 connection(s)
I20260619 01:00:14.818797 6859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:14.825652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6859
I20260619 01:00:14.825824 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:14.826300 388 external_mini_cluster.cc:1428] Setting key 08a3208eb58337a764449fbfcfcdcdde
Jun 19 01:00:14 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830814, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:14.838022 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:14.821334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52981 (local address 127.0.97.62:46621)
0619 01:00:14.821501 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:14.821505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:14.822615 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 01:00:14.824419 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:14.824425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:14.824508 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:14.824634 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:14.826193 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.827234 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.829303 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.829704 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.829797 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:14.833050 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:14.833055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:14.833058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:14.833088 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:14.835471 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:14.836145 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:14.836148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:14.836151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:14.836271 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:14.836677 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:14.836680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:14.836682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:14.836916 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:14.837019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:14.837638 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 01:00:14.837793 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:00:14.839068 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:14.821809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:52981)
0619 01:00:14.822458 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:14.822500 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:14.824161 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:14.824850 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:14.824859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:14.825378 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:14.826032 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:14.826050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.827400 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.827404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:14.829147 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:14.829167 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:14.829882 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:14.829886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:14.830083 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:14.830688 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:14.830751 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:14.832852 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:14.835607 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:14.835614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:14.835627 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:14.836051 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:14.836390 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:14.836394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:14.836397 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:14.836583 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:14.837018 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:14.837024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:14.837327 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 01:00:14.837554 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 01:00:14.837919 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":252,"threads_started":1}
I20260619 01:00:14.840531 6987 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:14.840867 6987 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:14.841682 6987 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:14.843026 6530 ts_manager.cc:194] Registered new tserver with Master: feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017)
I20260619 01:00:14.843815 6530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52981
I20260619 01:00:14.853773 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260619 01:00:14.880431 6720 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260619 01:00:14.899647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36749
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46621,127.0.97.61:36749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:15.216249 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:15.216532 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:15.216626 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:15.223685 6992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:15.223778 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:15.223821 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:15.223876 6992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:15.223915 6992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:15.233073 6992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46621,127.0.97.61:36749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
W20260619 01:00:15.233342 6992 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46621,127.0.97.61:36749'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 01:00:15.234284 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:15.235459 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:15.247114 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:15.247257 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:15.247653 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:15.248009 6992 server_base.cc:1034] running on GCE node
I20260619 01:00:15.248730 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:15.250022 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:15.251253 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781830815251194 us; error 39 us; skew 500 ppm
I20260619 01:00:15.251572 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:15.254518 6992 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 01:00:15.255873 6992 webserver.cc:460] Webserver started at http://127.0.97.61:33411/ using document root <none> and password file <none>
I20260619 01:00:15.256481 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:15.256603 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:15.256919 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:15.259819 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/instance:
uuid: "82516a363b0842a6953c69829d53cf91"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "c81cd3ec9c3d4ecb0888e5bcc2f245f3"
server_key_iv: "18fad7c9466b976976c0c9167fdf9dc6"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.260571 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal/instance:
uuid: "82516a363b0842a6953c69829d53cf91"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "c81cd3ec9c3d4ecb0888e5bcc2f245f3"
server_key_iv: "18fad7c9466b976976c0c9167fdf9dc6"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.265405 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 01:00:15.268679 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:15.269971 6992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:00:15.270149 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "82516a363b0842a6953c69829d53cf91"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "c81cd3ec9c3d4ecb0888e5bcc2f245f3"
server_key_iv: "18fad7c9466b976976c0c9167fdf9dc6"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.270356 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:15.292551 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:15.297032 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:15.297297 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:15.314291 6992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36749
I20260619 01:00:15.314309 7058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36749 every 8 connection(s)
I20260619 01:00:15.315635 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:00:15.317812 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 6992
I20260619 01:00:15.319936 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:15.328814 7059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:15.342200 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.319953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.0.97.61:36749)
0619 01:00:15.320721 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.320735 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.320780 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:00:15.322892 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.322920 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.323418 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.323827 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.325092 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.327081 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.328238 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.329187 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.329361 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.333730 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.333737 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.333747 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.333781 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.337476 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.338152 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.338166 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.338174 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.338368 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.338857 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.338861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.338864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.339503 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.339806 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.340449 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.341009 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":332,"threads_started":1}
I20260619 01:00:15.347004 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.331357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44022)
0619 01:00:15.332114 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.332160 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.332305 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.332666 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.332670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.333302 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.333521 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.333529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.334457 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.334465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.335904 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.335917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.336775 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.336782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.337013 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.338250 (+ 1237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.338270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.340973 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.344644 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.344656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.344664 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.345145 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.345515 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.345518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.345520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.345640 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.346197 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.346202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.346346 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.346489 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.346684 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":349,"threads_started":1}
I20260619 01:00:15.347085 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.331415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44022 (local address 127.0.97.62:46621)
0619 01:00:15.331655 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.331664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.332244 (+ 580us) server_negotiation.cc:407] Connection header received
0619 01:00:15.332417 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.332423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.332498 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.332605 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.333670 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.334332 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.336101 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.336572 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.336734 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.341171 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.341176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.341179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.341224 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.344487 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.345268 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.345273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.345276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.345414 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.345729 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.345732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.345734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.346080 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.346201 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.346621 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.346841 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:00:15.348475 7059 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:00:15.359333 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.349174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39960)
0619 01:00:15.349353 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.349372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.349459 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.349709 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.349712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.349983 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.350117 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.350124 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.351058 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.351062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.351897 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.351906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.352504 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.352508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.352637 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.353126 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.353140 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.354924 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.357476 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.357479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.357481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.357805 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.358100 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.358104 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.358106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.358231 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.358650 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.358653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.358815 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.358929 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.359113 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 01:00:15.359613 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.349206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39960 (local address 127.0.97.61:36749)
0619 01:00:15.349358 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.349362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.349434 (+ 72us) server_negotiation.cc:407] Connection header received
0619 01:00:15.349535 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.349539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.349594 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.349668 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.350223 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.350944 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.352003 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.352394 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.352477 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.355060 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.355064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.355067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.355093 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.357377 (+ 2284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.357895 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.357898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.357900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.358000 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.358314 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.358317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.358318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.358532 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.358681 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.359003 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.359241 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":44}
I20260619 01:00:15.363857 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Bootstrap starting.
I20260619 01:00:15.366689 7059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:15.367672 7059 log.cc:824] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:15.368419 6854 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
I20260619 01:00:15.370498 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: No bootstrap required, opened a new log
I20260619 01:00:15.377417 7059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } }
I20260619 01:00:15.377768 7059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:15.377887 7059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82516a363b0842a6953c69829d53cf91, State: Initialized, Role: FOLLOWER
I20260619 01:00:15.378433 7059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } }
I20260619 01:00:15.379236 7065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } } }
I20260619 01:00:15.379552 7065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:15.380276 7059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 01:00:15.389235 7076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:00:15.389379 7076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:00:15.391395 6992 master_runner.cc:428] Detected that this master 82516a363b0842a6953c69829d53cf91 is joining an existing cluster
I20260619 01:00:15.391495 6992 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:36749
I20260619 01:00:15.412791 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.395686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.0.97.62:46621)
0619 01:00:15.395864 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.395871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.396581 (+ 710us) server_negotiation.cc:407] Connection header received
0619 01:00:15.396840 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.396846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.396920 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.396987 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.398381 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.399511 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.401512 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.402131 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.402239 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.405353 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.405361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.405367 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.405409 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.409364 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.410159 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.410165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.410169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.410304 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.410694 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.410701 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.410704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.411081 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.411236 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.411897 (+ 661us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.412414 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:15.414572 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.395719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44026)
0619 01:00:15.396481 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.396500 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.396688 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.397186 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.397191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.397812 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.398205 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.398217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.399687 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.399692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.401323 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.401337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.402529 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.402536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.402753 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.403822 (+ 1069us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.403854 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.405102 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.409515 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.409523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.409530 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.409982 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.410429 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.410434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.410437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.410585 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.411297 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.411302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.411576 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.413637 (+ 2061us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.414030 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"spinlock_wait_cycles":26368,"thread_start_us":7005,"threads_started":1}
I20260619 01:00:15.420542 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.395719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.0.97.61:36749)
0619 01:00:15.395922 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.395931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.403221 (+ 7290us) server_negotiation.cc:407] Connection header received
0619 01:00:15.403336 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.403342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.403449 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.403575 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.405000 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.406149 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.408263 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.408741 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.408882 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.411302 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.411311 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.411319 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.411360 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.415476 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.416330 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.416335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.416338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.416496 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.416958 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.416965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.416972 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.417406 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.417597 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.419884 (+ 2287us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.420119 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:00:15.421633 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.396292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39962)
0619 01:00:15.403082 (+ 6790us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.403102 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.403226 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.403609 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.403616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.404491 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.404849 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.404863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.406331 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.406339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.408049 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.408068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.408921 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.408927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.409131 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.409787 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.409818 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.411077 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.415627 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.415631 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.415635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.416198 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.416647 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.416654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.416661 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.416836 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.417681 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.417705 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.418280 (+ 575us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.420785 (+ 2505us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.421075 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6657,"mutex_wait_us":54,"thread_start_us":6928,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:00:15.440533 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.422075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.0.97.61:36749)
0619 01:00:15.423775 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.423787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.423811 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:00:15.423908 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.423915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.424014 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.424137 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.425546 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.426908 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.428758 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.429305 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.429465 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.433570 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.433576 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.433581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.433615 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.437186 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.438088 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.438094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.438097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.438243 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.438755 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.438760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.438763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.439146 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.439298 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.439910 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.440136 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:00:15.440778 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.422039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39966)
0619 01:00:15.422788 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.422839 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.423003 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.424197 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.424205 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.425097 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.425361 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.425373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.427049 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.427055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.428617 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.428632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.429488 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.429497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.429714 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.430339 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.430363 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.433342 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.437331 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.437346 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.437356 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.437957 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.438363 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.438371 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.438377 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.438588 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.439304 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.439316 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.439569 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.439807 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.440165 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":295,"threads_started":1}
I20260619 01:00:15.443128 6530 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:36749: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "82516a363b0842a6953c69829d53cf91"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 36749
}
attrs {
promote: true
}
}
dest_uuid: "82345d268ed64aa3957b1563bd6f8d75"
cas_config_opid_index: -1
I20260619 01:00:15.444342 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } }
I20260619 01:00:15.453140 7088 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 82516a363b0842a6953c69829d53cf91 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } }
I20260619 01:00:15.454909 7034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:15.457649 7088 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:36749
I20260619 01:00:15.458587 6992 master.cc:537] Master@127.0.97.61:36749 shutting down...
I20260619 01:00:15.458418 7089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } }
I20260619 01:00:15.459890 7089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:15.460316 7034 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Refusing update from remote peer 82345d268ed64aa3957b1563bd6f8d75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 01:00:15.462093 7088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:00:15.466506 6518 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260619 01:00:15.469775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38947
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:00:15.473239 6992 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:15.473649 6992 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:15.473883 6992 tablet_replica.cc:330] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: stopping tablet replica
W20260619 01:00:15.474692 6518 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 -> Peer 82516a363b0842a6953c69829d53cf91 (127.0.97.61:36749): Couldn't send request to peer 82516a363b0842a6953c69829d53cf91. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 01:00:15.481165 6992 master.cc:552] Master@127.0.97.61:36749 shutdown complete.
I20260619 01:00:15.481290 6992 master_runner.cc:305] Clearing existing system tablet
I20260619 01:00:15.481772 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:15.481946 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:15.485384 6992 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:15.487172 7094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:15.487461 6992 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:15.487869 6992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:15.488098 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "82516a363b0842a6953c69829d53cf91"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "c81cd3ec9c3d4ecb0888e5bcc2f245f3"
server_key_iv: "18fad7c9466b976976c0c9167fdf9dc6"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.493374 6992 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:15.500422 6992 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 01:00:15.500700 6992 log.cc:1196] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 01:00:15.501206 6992 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Deleting consensus metadata
I20260619 01:00:15.501526 6992 master_runner.cc:315] Copying system tablet from 127.0.97.62:46621
I20260619 01:00:15.504212 6992 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46621
I20260619 01:00:15.534124 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.505126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.0.97.62:46621)
0619 01:00:15.505349 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:15.505356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:15.519397 (+ 14041us) server_negotiation.cc:407] Connection header received
0619 01:00:15.519611 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:15.519618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:15.519731 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:15.519874 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:15.520957 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.522094 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.523905 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.524591 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.524727 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.527121 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:15.527130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:15.527138 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:15.527187 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:15.530960 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.531843 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.531849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.531852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.532009 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:15.532473 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:15.532480 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:15.532483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:15.532893 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:15.533066 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:15.533495 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 01:00:15.533733 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:00:15.534158 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:15.505081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44040)
0619 01:00:15.519220 (+ 14139us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:15.519253 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:15.519434 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:15.519912 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:15.519917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:15.520441 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:15.520725 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.520739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.522263 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.522269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.523734 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:15.523750 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:15.524767 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:15.524773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:15.524967 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:15.525728 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:15.525752 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:15.526868 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:15.531187 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.531193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.531197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.531692 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.532131 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:15.532136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:15.532140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:15.532326 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:15.533021 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:15.533026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:15.533197 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 01:00:15.533336 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 01:00:15.533675 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13975,"thread_start_us":13834,"threads_started":1}
I20260619 01:00:15.535360 6551 tablet_copy_service.cc:140] P 82345d268ed64aa3957b1563bd6f8d75: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 82516a363b0842a6953c69829d53cf91 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44040)
I20260619 01:00:15.535672 6551 tablet_copy_service.cc:161] P 82345d268ed64aa3957b1563bd6f8d75: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 82516a363b0842a6953c69829d53cf91 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44040: session id = 82516a363b0842a6953c69829d53cf91-00000000000000000000000000000000
I20260619 01:00:15.540094 6551 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 01:00:15.542384 6992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 01:00:15.549587 6992 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: tablet copy: Starting download of 0 data blocks...
I20260619 01:00:15.549947 6992 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: tablet copy: Starting download of 1 WAL segments...
I20260619 01:00:15.552831 6992 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 01:00:15.557337 6551 tablet_copy_service.cc:342] P 82345d268ed64aa3957b1563bd6f8d75: Request end of tablet copy session 82516a363b0842a6953c69829d53cf91-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44040
I20260619 01:00:15.557566 6551 tablet_copy_service.cc:433] P 82345d268ed64aa3957b1563bd6f8d75: ending tablet copy session 82516a363b0842a6953c69829d53cf91-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 82516a363b0842a6953c69829d53cf91
I20260619 01:00:15.570218 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:15.573200 7101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:15.577553 7102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:15.585085 6992 server_base.cc:1034] running on GCE node
W20260619 01:00:15.585572 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:15.585994 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:15.586640 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:15.587924 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781830815587870 us; error 49 us; skew 500 ppm
I20260619 01:00:15.588157 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:15.590483 6992 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260619 01:00:15.603201 6992 webserver.cc:460] Webserver started at http://127.0.97.61:42125/ using document root <none> and password file <none>
I20260619 01:00:15.603621 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:15.603734 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:15.607372 6992 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260619 01:00:15.609469 7111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:15.610008 6992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260619 01:00:15.610162 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "82516a363b0842a6953c69829d53cf91"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "c81cd3ec9c3d4ecb0888e5bcc2f245f3"
server_key_iv: "18fad7c9466b976976c0c9167fdf9dc6"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.610347 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:15.652215 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:15.656661 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:15.703764 6992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36749
I20260619 01:00:15.703951 7162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36749 every 8 connection(s)
I20260619 01:00:15.704828 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:00:15.710156 7163 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:00:15.714183 7163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Bootstrap starting.
I20260619 01:00:15.735002 7163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:00:15.735589 7163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Bootstrap complete.
I20260619 01:00:15.737143 7163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } }
I20260619 01:00:15.737401 7163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82516a363b0842a6953c69829d53cf91, State: Initialized, Role: LEARNER
I20260619 01:00:15.737690 7163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } }
I20260619 01:00:15.738495 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } }
I20260619 01:00:15.738819 7163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:15.738929 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:00:15.747637 7177 catalog_manager.cc:1259] Loaded cluster ID: 288596af42a54b8d8ce48c50288d05f8
I20260619 01:00:15.747766 7177 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: loading cluster ID for follower catalog manager: success
I20260619 01:00:15.750386 7177 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: acquiring CA information for follower catalog manager: success
I20260619 01:00:15.752228 7177 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:00:15.846266 6987 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
W20260619 01:00:15.900020 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:15.900341 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:15.900506 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:15.907784 7092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:15.907917 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:15.908028 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:15.908131 7092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:15.908213 7092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:15.918223 7092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:38947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:15.919616 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:15.920845 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:15.932559 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:15.932500 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:15.933104 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:15.933563 7092 server_base.cc:1034] running on GCE node
I20260619 01:00:15.934300 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:15.935775 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:15.937002 7092 hybrid_clock.cc:648] HybridClock initialized: now 1781830815936932 us; error 42 us; skew 500 ppm
I20260619 01:00:15.937342 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:15 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:15.944658 7092 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 01:00:15.946151 7092 webserver.cc:460] Webserver started at http://127.0.97.60:36735/ using document root <none> and password file <none>
I20260619 01:00:15.946933 7092 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:15.947075 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:15.947386 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:15.950314 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/instance:
uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "755db5036c1230c2f02ea02d6caeb4c3"
server_key_iv: "35916969a1ce34c3d142b271b25b17bd"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.951112 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal/instance:
uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "755db5036c1230c2f02ea02d6caeb4c3"
server_key_iv: "35916969a1ce34c3d142b271b25b17bd"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.956089 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 01:00:15.959395 7192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:15.960691 7092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:00:15.960878 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "755db5036c1230c2f02ea02d6caeb4c3"
server_key_iv: "35916969a1ce34c3d142b271b25b17bd"
server_key_version: "encryptionkey@0"
I20260619 01:00:15.961073 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:15.974445 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:15.979429 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:15.979717 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:15.998272 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38947
I20260619 01:00:15.998296 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38947 every 8 connection(s)
I20260619 01:00:15.999697 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:00:16.003700 7244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:16.009426 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 7092
I20260619 01:00:16.009968 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:00:16.030138 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.017686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.0.97.61:36749)
0619 01:00:16.022786 (+ 5100us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.022793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.023182 (+ 389us) server_negotiation.cc:407] Connection header received
0619 01:00:16.023264 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.023270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.023384 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.023532 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:16.024175 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.025152 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.027461 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.028966 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.029132 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.029383 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.029651 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.029820 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4911,"thread_start_us":4786,"threads_started":1}
I20260619 01:00:16.030177 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.017494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39974)
0619 01:00:16.022905 (+ 5411us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.022940 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.023060 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.023750 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.023757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.023787 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:16.024022 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.024031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.025290 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.025295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.027308 (+ 2013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.027319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.029164 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.029173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.029441 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.029503 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.029621 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.029799 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5298,"thread_start_us":5190,"threads_started":1}
I20260619 01:00:16.031504 7138 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 01:00:16.032141 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82345d268ed64aa3957b1563bd6f8d75. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } }
I20260619 01:00:16.032384 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:00:16.034044 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.012814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.0.97.62:46621)
0619 01:00:16.013017 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.013024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.013655 (+ 631us) server_negotiation.cc:407] Connection header received
0619 01:00:16.014574 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.014582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.014658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.014797 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.016541 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.017562 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.020673 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.021588 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.021723 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.027208 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.027217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.027225 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.027274 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.030780 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.031517 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.031524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.031531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.031657 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.032236 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.032242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.032245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.032578 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.032676 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.033531 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.033730 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:00:16.035262 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.011560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.0.97.60:38947)
0619 01:00:16.012349 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.012358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.012395 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:00:16.014321 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.014341 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.014613 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.014834 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.016646 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.019178 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.021588 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.022756 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.022917 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.028015 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.028025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.028041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.028086 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.032301 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.033052 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.033069 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.033072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.033208 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.033542 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.033547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.033550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.033851 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.034164 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.034440 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.034676 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":615,"thread_start_us":337,"threads_started":1}
I20260619 01:00:16.035282 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.012632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44056)
0619 01:00:16.013495 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.013534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.014335 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.014866 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.014872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.015851 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.016347 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.016372 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.017758 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.017764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.020436 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.020459 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.022155 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.022183 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.023154 (+ 971us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.023922 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.023956 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.027018 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.030910 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.030923 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.030943 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.031394 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.031848 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.031853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.031856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.032095 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.032758 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.032763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.033120 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.033462 (+ 342us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.033939 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":593,"thread_start_us":447,"threads_started":1}
I20260619 01:00:16.036767 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:00:16.048719 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.037400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39980)
0619 01:00:16.037579 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.037599 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.037703 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.038018 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.038022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.038356 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.038475 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.038482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.039543 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.039548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.040517 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.040527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.041152 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.041156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.041327 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.041920 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.041936 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.043887 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.046914 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.046917 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.046920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.047264 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.047579 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.047582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.047584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.047694 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.048106 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.048109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.048214 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.048315 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.048503 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 01:00:16.048794 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.037463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.0.97.61:36749)
0619 01:00:16.037626 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.037630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.037644 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:00:16.037816 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.037821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.037895 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.037984 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.038594 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.039410 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.040633 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.041029 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.041129 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.044030 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.044034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.044037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.044081 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.046799 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.047357 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.047360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.047362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.047485 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.047778 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.047781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.047783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.048003 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.048112 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.048394 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.048562 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:00:16.049592 7244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830815, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:00:16.060930 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.050189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49330)
0619 01:00:16.050394 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.050418 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.050545 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.050849 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.050853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.051186 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.051309 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.051316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.052196 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.052204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.052984 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.052993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.053643 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.053648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.053886 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.054313 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.054328 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.056205 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.058667 (+ 2462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.058675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.058680 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.059021 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.059364 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.059371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.059375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.059509 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.059978 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.059982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.060119 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.060247 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.060502 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":20}
I20260619 01:00:16.060930 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.050246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.0.97.60:38947)
0619 01:00:16.050394 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.050401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.050487 (+ 86us) server_negotiation.cc:407] Connection header received
0619 01:00:16.050650 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.050655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.050754 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.050837 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.051430 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.052082 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.053117 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.053501 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.053637 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.056368 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.056374 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.056379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.056412 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.058524 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.059135 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.059142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.059147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.059265 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.059617 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.059624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.059626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.059870 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.060010 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.060331 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.060628 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":154,"server-negotiator.queue_time_us":46}
I20260619 01:00:16.065755 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Bootstrap starting.
I20260619 01:00:16.069203 7244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:16.070278 7244 log.cc:824] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:16.072715 7244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: No bootstrap required, opened a new log
I20260619 01:00:16.079542 7244 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } }
I20260619 01:00:16.079860 7244 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:16.079977 7244 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2bbf30bacc44d58998cc8b53bcf05b4, State: Initialized, Role: FOLLOWER
I20260619 01:00:16.080610 7244 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } }
I20260619 01:00:16.081441 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } } }
I20260619 01:00:16.081774 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:16.082960 7244 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:16.089797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6592
W20260619 01:00:16.093510 7264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:00:16.093745 7264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:00:16.096388 7092 master_runner.cc:428] Detected that this master d2bbf30bacc44d58998cc8b53bcf05b4 is joining an existing cluster
I20260619 01:00:16.096514 7092 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:38947
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 01:00:16.113557 7266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.091119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.97.1:37835)
0619 01:00:16.092160 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.092169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.092195 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:00:16.092267 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.092275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.092528 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.093965 (+ 1437us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.094314 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.095672 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.097559 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.098424 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.104231 (+ 5807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.104502 (+ 271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.104514 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.104531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.104585 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.108468 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.109671 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.109701 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.109707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.109855 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.110310 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.110315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.110319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.110829 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.112295 (+ 1466us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.112461 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.112731 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":752,"threads_started":1}
I20260619 01:00:16.115515 6715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:16.133160 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.101329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44070 (local address 127.0.97.62:46621)
0619 01:00:16.101580 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.101589 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.115558 (+ 13969us) server_negotiation.cc:407] Connection header received
0619 01:00:16.115987 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.115995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.116102 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.116189 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.118012 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.119530 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.122052 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.122693 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.122832 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.125564 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.125571 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.125576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.125619 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.129756 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.130640 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.130650 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.130656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.130844 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.131340 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.131347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.131354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.131756 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.131895 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.132492 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.132724 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:00:16.134574 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.100864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44070)
0619 01:00:16.115430 (+ 14566us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.115452 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.115779 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.116461 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.116468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.117344 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.117798 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.117815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.119781 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.119790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.121889 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.121903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.122888 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.122893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.123083 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.123894 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.123926 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.125333 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.129915 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.129925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.129929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.130484 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.130999 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.131006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.131013 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.131211 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.131993 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.131999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.132206 (+ 207us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.133696 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.134038 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14396,"thread_start_us":17048,"threads_started":1}
I20260619 01:00:16.143517 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.101118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39986 (local address 127.0.97.61:36749)
0619 01:00:16.101327 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.101334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.118972 (+ 17638us) server_negotiation.cc:407] Connection header received
0619 01:00:16.119385 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.119394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.119501 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.119605 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.121489 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.122912 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.125583 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.126236 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.126330 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.135116 (+ 8786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.135126 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.135130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.135173 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.139077 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.139918 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.139923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.139926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.140083 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.140550 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.140554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.140557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.141662 (+ 1105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.141806 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.142644 (+ 838us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.142991 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:00:16.145258 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.101136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:39986)
0619 01:00:16.118852 (+ 17716us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.118874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.119075 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.119874 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.119882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.120812 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.121279 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.121295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.123156 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.123164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.125383 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.125402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.127978 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.127990 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.128227 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.129140 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.129171 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.134842 (+ 5671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.139270 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.139276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.139282 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.139758 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.140230 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.140234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.140240 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.140414 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.141921 (+ 1507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.141930 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.142237 (+ 307us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.144129 (+ 1892us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.144581 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17587,"thread_start_us":17351,"threads_started":1}
I20260619 01:00:16.146088 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.119075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.0.97.60:38947)
0619 01:00:16.119250 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.119256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.126001 (+ 6745us) server_negotiation.cc:407] Connection header received
0619 01:00:16.126213 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.126218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.126287 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.126342 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.132647 (+ 6305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.133769 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.135755 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.136490 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.136582 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.139460 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.139466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.139471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.139508 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.143208 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.143957 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.143962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.143967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.144114 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.144494 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.144498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.144502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.144846 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.144999 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.145535 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.145741 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:00:16.146904 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.119236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49342)
0619 01:00:16.125918 (+ 6682us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.125937 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.126092 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.126510 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.126516 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.132147 (+ 5631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.132498 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.132510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.133982 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.133991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.135609 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.135623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.136876 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.136883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.137081 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.138072 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.138099 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.139261 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.143352 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.143356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.143360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.143831 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.144249 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.144253 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.144256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.144384 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.145088 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.145093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.145286 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.146147 (+ 861us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.146412 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6405,"mutex_wait_us":3,"thread_start_us":6506,"threads_started":1}
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:00:16.160054 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.141070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49352)
0619 01:00:16.141312 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.141333 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.141443 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.147570 (+ 6127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.147578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.148176 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.148379 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.148390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.149889 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.149895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.151165 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.151177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.151911 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.151917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.152105 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.152705 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.152722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.154670 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.157753 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.157759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.157763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.158130 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.158484 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.158490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.158493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.158623 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.159189 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.159194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.159378 (+ 184us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.159555 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.159824 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 01:00:16.160844 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.141133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.0.97.60:38947)
0619 01:00:16.147174 (+ 6041us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.147180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.147197 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:00:16.147285 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.147290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.147403 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.147502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.148537 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.149582 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.151334 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.151759 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.151896 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.154916 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.154921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.154924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.154951 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.157597 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.158247 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.158253 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.158257 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.158376 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.158789 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.158795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.158798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.159072 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.159222 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.160383 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.160611 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5937,"thread_start_us":6955,"threads_started":1}
I20260619 01:00:16.162137 6530 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:38947: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 38947
}
attrs {
promote: true
}
}
dest_uuid: "82345d268ed64aa3957b1563bd6f8d75"
cas_config_opid_index: 5
I20260619 01:00:16.163058 6530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } }
I20260619 01:00:16.164733 7253 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:16.164987 7253 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } }
I20260619 01:00:16.165290 7249 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.166414 7253 leader_election.cc:288] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621), 82516a363b0842a6953c69829d53cf91 (127.0.97.61:36749)
I20260619 01:00:16.167788 6541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82345d268ed64aa3957b1563bd6f8d75" is_pre_election: true
I20260619 01:00:16.170459 7249 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:38947
I20260619 01:00:16.170928 7138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82516a363b0842a6953c69829d53cf91" is_pre_election: true
I20260619 01:00:16.172011 7196 leader_election.cc:302] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2bbf30bacc44d58998cc8b53bcf05b4; no voters: 82345d268ed64aa3957b1563bd6f8d75, 82516a363b0842a6953c69829d53cf91
I20260619 01:00:16.172969 7253 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 01:00:16.174568 7138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Refusing update from remote peer 82345d268ed64aa3957b1563bd6f8d75: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 01:00:16.178315 7092 master.cc:537] Master@127.0.97.60:38947 shutting down...
I20260619 01:00:16.189828 7276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 01:00:16.190397 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.190920 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:16.201067 7092 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:16.201447 7092 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:16.201628 7092 tablet_replica.cc:330] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: stopping tablet replica
I20260619 01:00:16.208886 7092 master.cc:552] Master@127.0.97.60:38947 shutdown complete.
I20260619 01:00:16.209030 7092 master_runner.cc:305] Clearing existing system tablet
I20260619 01:00:16.209540 7092 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:16.209703 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:16.209911 7138 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.212128 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.211637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49356)
0619 01:00:16.211806 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.211946 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 01:00:16.212524 6518 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 -> Peer d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60:38947): Couldn't send request to peer d2bbf30bacc44d58998cc8b53bcf05b4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 01:00:16.213958 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: true } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.214269 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:00:16.214677 7277 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: attempting to promote NON_VOTER 82516a363b0842a6953c69829d53cf91 to VOTER
I20260619 01:00:16.215575 7277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } }
I20260619 01:00:16.217741 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.217362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49360)
0619 01:00:16.217503 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.217591 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 01:00:16.219388 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.219065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49374)
0619 01:00:16.219180 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.219256 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38947: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 01:00:16.219671 6518 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 -> Peer d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60:38947): Couldn't send request to peer d2bbf30bacc44d58998cc8b53bcf05b4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:38947: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 01:00:16.220139 7138 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 LEARNER]: Refusing update from remote peer 82345d268ed64aa3957b1563bd6f8d75: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 01:00:16.220688 7277 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 01:00:16.222031 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.001s
I20260619 01:00:16.223161 7249 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 82516a363b0842a6953c69829d53cf91 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.224128 7137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 82516a363b0842a6953c69829d53cf91 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.228037 7166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.228284 7166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:16.228241 7276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.228613 7276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:16.229058 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.229383 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:16.230309 7282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:16.230697 7092 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:16.230980 7092 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.004s
I20260619 01:00:16.231094 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "755db5036c1230c2f02ea02d6caeb4c3"
server_key_iv: "35916969a1ce34c3d142b271b25b17bd"
server_key_version: "encryptionkey@0"
I20260619 01:00:16.235988 7092 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:16.243037 7092 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 01:00:16.243292 7092 log.cc:1196] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 01:00:16.243811 7092 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Deleting consensus metadata
I20260619 01:00:16.244108 7092 master_runner.cc:315] Copying system tablet from 127.0.97.62:46621
I20260619 01:00:16.264155 7092 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46621
I20260619 01:00:16.276280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6592
I20260619 01:00:16.281081 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.264938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44074)
0619 01:00:16.265597 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.265622 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.265777 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.266147 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.266152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.266686 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.266979 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.266997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.268450 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.268456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.270009 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.270022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.271050 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.271056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.271225 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.271940 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.271961 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.272979 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.277300 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.277308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.277315 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.277883 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.278332 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.278339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.278346 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.278510 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.279726 (+ 1216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.279738 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.279943 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.280091 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.280468 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":419,"threads_started":1}
I20260619 01:00:16.282152 6722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.265003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.0.97.62:46621)
0619 01:00:16.265247 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.265253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.265728 (+ 475us) server_negotiation.cc:407] Connection header received
0619 01:00:16.265912 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.265917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.265995 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.266099 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.267150 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.268296 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.270217 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.270883 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.271012 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.273205 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.273213 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.273221 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.273268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.277148 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.278028 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.278036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.278043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.278186 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.278635 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.278642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.278648 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.279499 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.279631 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.281451 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.281718 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 01:00:16.282886 6551 tablet_copy_service.cc:140] P 82345d268ed64aa3957b1563bd6f8d75: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d2bbf30bacc44d58998cc8b53bcf05b4 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44074)
I20260619 01:00:16.283061 6551 tablet_copy_service.cc:161] P 82345d268ed64aa3957b1563bd6f8d75: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d2bbf30bacc44d58998cc8b53bcf05b4 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44074: session id = d2bbf30bacc44d58998cc8b53bcf05b4-00000000000000000000000000000000
I20260619 01:00:16.287717 6551 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 01:00:16.289762 7092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 01:00:16.296717 7092 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: tablet copy: Starting download of 0 data blocks...
I20260619 01:00:16.297022 7092 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: tablet copy: Starting download of 1 WAL segments...
I20260619 01:00:16.297325 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:37835
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46621
--webserver_port=38635
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:00:16.301901 7092 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 01:00:16.305573 6551 tablet_copy_service.cc:342] P 82345d268ed64aa3957b1563bd6f8d75: Request end of tablet copy session d2bbf30bacc44d58998cc8b53bcf05b4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44074
I20260619 01:00:16.305716 6551 tablet_copy_service.cc:433] P 82345d268ed64aa3957b1563bd6f8d75: ending tablet copy session d2bbf30bacc44d58998cc8b53bcf05b4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d2bbf30bacc44d58998cc8b53bcf05b4
I20260619 01:00:16.315914 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:16.318519 7290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:16.324721 7291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:16.329365 7092 server_base.cc:1034] running on GCE node
W20260619 01:00:16.330938 7293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:16.331360 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:16.331996 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:16.333294 7092 hybrid_clock.cc:648] HybridClock initialized: now 1781830816333248 us; error 40 us; skew 500 ppm
I20260619 01:00:16.333516 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830816, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:16.335410 7092 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260619 01:00:16.343701 7092 webserver.cc:460] Webserver started at http://127.0.97.60:45161/ using document root <none> and password file <none>
I20260619 01:00:16.344002 7092 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:16.344121 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:16.346489 7092 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 01:00:16.348201 7300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:16.348574 7092 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 01:00:16.348702 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
format_stamp: "Formatted at 2026-06-19 01:00:15 on dist-test-slave-3rxd"
server_key: "755db5036c1230c2f02ea02d6caeb4c3"
server_key_iv: "35916969a1ce34c3d142b271b25b17bd"
server_key_version: "encryptionkey@0"
I20260619 01:00:16.348881 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:16.361886 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:16.365226 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:16.396965 7092 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:38947
I20260619 01:00:16.397353 7351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:38947 every 8 connection(s)
I20260619 01:00:16.398243 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:00:16.405195 7352 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:00:16.410835 7352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Bootstrap starting.
I20260619 01:00:16.425550 7352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:00:16.425984 7352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: Bootstrap complete.
I20260619 01:00:16.426841 7352 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } }
I20260619 01:00:16.427094 7352 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d2bbf30bacc44d58998cc8b53bcf05b4, State: Initialized, Role: LEARNER
I20260619 01:00:16.427317 7352 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } }
I20260619 01:00:16.428094 7355 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.428432 7352 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:16.428462 7355 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:00:16.445160 7366 catalog_manager.cc:1259] Loaded cluster ID: 288596af42a54b8d8ce48c50288d05f8
I20260619 01:00:16.445278 7366 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: loading cluster ID for follower catalog manager: success
I20260619 01:00:16.447525 7366 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: acquiring CA information for follower catalog manager: success
I20260619 01:00:16.449450 7366 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 01:00:16.676345 7289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:16.676651 7289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:16.676817 7289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:16.684044 7289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:16.684177 7289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:16.684360 7289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:16.693856 7289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:37835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38635
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:16.695619 7289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:16.696920 7289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:16.711225 7371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:16.711154 7372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:16.711736 7374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:16.712107 7289 server_base.cc:1034] running on GCE node
I20260619 01:00:16.712780 7289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:16.713716 7289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:16.714926 7289 hybrid_clock.cc:648] HybridClock initialized: now 1781830816714870 us; error 51 us; skew 500 ppm
I20260619 01:00:16.715231 7289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830816, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:16.718284 7289 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:00:16.719791 7289 webserver.cc:460] Webserver started at http://127.0.97.1:38635/ using document root <none> and password file <none>
I20260619 01:00:16.720451 7289 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:16.720614 7289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:16.725797 7289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 01:00:16.728804 7381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:16.730387 7289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:00:16.730558 7289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "cb803022fd5945729575948839a387d5"
format_stamp: "Formatted at 2026-06-19 01:00:13 on dist-test-slave-3rxd"
server_key: "f89c5d488f0aa375313a3f2a4c1d5136"
server_key_iv: "cea1b4f70d3365709ebaa3992fefb45a"
server_key_version: "encryptionkey@0"
I20260619 01:00:16.731554 7289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:16.740516 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.732443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49390 (local address 127.0.97.60:38947)
0619 01:00:16.735161 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.735166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.735182 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:00:16.735238 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.735241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.735306 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.735435 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:16.735865 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.736824 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.738397 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.739479 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.739622 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.739816 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.740043 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.740182 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2610,"thread_start_us":2495,"threads_started":1}
I20260619 01:00:16.740523 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.732303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49390)
0619 01:00:16.732691 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.732711 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.732844 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.735458 (+ 2614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.735463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.735493 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:16.735707 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.735717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.736978 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.736984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.738262 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.738279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.739659 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.739665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.739858 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.739898 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.740020 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.740182 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":236,"threads_started":1}
I20260619 01:00:16.741696 7327 raft_consensus.cc:1203] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 01:00:16.742327 7355 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82345d268ed64aa3957b1563bd6f8d75. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:16.742640 7355 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: LEARNER
I20260619 01:00:16.750576 7289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:16.754403 7289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:16.754683 7289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:16.755649 7289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:16.756899 7289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:16.757026 7289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:16.757151 7289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:16.757236 7289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:16.792030 7289 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37835
I20260619 01:00:16.792093 7496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37835 every 8 connection(s)
I20260619 01:00:16.793471 7289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:16.794437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 7289
I20260619 01:00:16.794837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6726
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830816, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830816, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260619 01:00:16.830523 7501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.796532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.0.97.2:38023)
0619 01:00:16.797177 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.797185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.805489 (+ 8304us) server_negotiation.cc:407] Connection header received
0619 01:00:16.805797 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.805806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.806054 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.806201 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.808398 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.809845 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.812547 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.813362 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.813460 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.821162 (+ 7702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.821179 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.821197 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.821242 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.826392 (+ 5150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.827300 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.827311 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.827317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.827468 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.827958 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.827964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.827967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.828407 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.828672 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.829385 (+ 713us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.830098 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":611,"threads_started":1}
I20260619 01:00:16.832064 6849 generic_service.cc:196] Checking for leaks (request via RPC)
Jun 19 01:00:16 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830816, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:00:16.833169 7502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.797035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47897 (local address 127.0.97.62:46621)
0619 01:00:16.797491 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.797497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.805142 (+ 7645us) server_negotiation.cc:407] Connection header received
0619 01:00:16.805374 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.805379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.805466 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.805540 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.812696 (+ 7156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.813812 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.815890 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.816629 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.816766 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.825151 (+ 8385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.825157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.825164 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.825202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.828323 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.829693 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.829699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.829702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.829869 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.831739 (+ 1870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.831744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.831750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.832111 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.832243 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.832612 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.832829 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":321,"threads_started":1}
I20260619 01:00:16.834069 7504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.798225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:47897)
0619 01:00:16.805037 (+ 6812us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.805055 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.805231 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.805820 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.805823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.806448 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.812536 (+ 6088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.812547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.813960 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.813965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.815704 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.815721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.818052 (+ 2331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.818059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.818379 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.819542 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.819600 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.824619 (+ 5019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.828728 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.828732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.828734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.829333 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.830189 (+ 856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.830194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.830198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.830401 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.832207 (+ 1806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.832212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.832346 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.832504 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.832888 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6669,"thread_start_us":6512,"threads_started":1}
I20260619 01:00:16.834367 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.797865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49327 (local address 127.0.97.60:38947)
0619 01:00:16.798041 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.798048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.798538 (+ 490us) server_negotiation.cc:407] Connection header received
0619 01:00:16.802050 (+ 3512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.802060 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.802170 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.802259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.807488 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.808678 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.816981 (+ 8303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.817634 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.817771 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.825356 (+ 7585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.825362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.825365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.825410 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.828431 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.830205 (+ 1774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.830211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.830214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.830346 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.830822 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.830827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.830830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.831105 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.831265 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.833150 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.833707 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:16.835568 7498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:16.835937 7498 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:16.835934 7503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.797260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.97.1:49327)
0619 01:00:16.798383 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.798433 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.801660 (+ 3227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.802927 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.802943 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.803896 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.807243 (+ 3347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.807271 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.810042 (+ 2771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.810047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.812266 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.812280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.818885 (+ 6605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.818892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.819137 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.820261 (+ 1124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.820283 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.825112 (+ 4829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.828620 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.828629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.828646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.830079 (+ 1433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.830464 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.830469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.830473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.830650 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.831246 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.831253 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.831617 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.831970 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.835238 (+ 3268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":1058,"threads_started":1}
I20260619 01:00:16.836922 7498 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:16.838972 7497 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38947
I20260619 01:00:16.838982 6530 ts_manager.cc:194] Re-registered known tserver with Master: cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:16.839139 7497 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:16.839622 7497 heartbeater.cc:511] Master 127.0.97.60:38947 requested a full tablet report, sending...
I20260619 01:00:16.840122 6530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47897
I20260619 01:00:16.841991 7317 ts_manager.cc:194] Registered new tserver with Master: cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:16.843032 7507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.806257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.97.1:60195)
0619 01:00:16.817933 (+ 11676us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:16.817953 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:16.818097 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:16.818687 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:16.818693 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:16.819309 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:16.820073 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.820085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.821632 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.821640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.823928 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:16.823943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.829587 (+ 5644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.829595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:16.829815 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.830790 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:16.830810 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:16.835944 (+ 5134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:16.840240 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.840245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.840248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.840745 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.841250 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:16.841254 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:16.841258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:16.841477 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:16.842072 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:16.842077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:16.842213 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 01:00:16.842343 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 01:00:16.842627 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11545,"thread_start_us":11385,"threads_started":1}
I20260619 01:00:16.843753 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:16.807076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60195 (local address 127.0.97.61:36749)
0619 01:00:16.817482 (+ 10406us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:16.817487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:16.818060 (+ 573us) server_negotiation.cc:407] Connection header received
0619 01:00:16.818219 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:16.818224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:16.818314 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:16.818441 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:16.820247 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.821410 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.824093 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:16.829420 (+ 5327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:16.829550 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:16.836205 (+ 6655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:16.836212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:16.836216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:16.836256 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:16.840086 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.840955 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.840960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.840963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.841107 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:16.841622 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:16.841626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:16.841628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:16.841931 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:16.843253 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:16.843365 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 01:00:16.843510 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10280,"thread_start_us":10176,"threads_started":1}
I20260619 01:00:16.844566 7499 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36749
I20260619 01:00:16.844681 7499 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:16.844950 7499 heartbeater.cc:511] Master 127.0.97.61:36749 requested a full tablet report, sending...
I20260619 01:00:16.846369 7128 ts_manager.cc:194] Registered new tserver with Master: cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:16.931694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6726
I20260619 01:00:16.947708 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:38023
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46621
--webserver_port=44091
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:17.270550 7509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:17.270846 7509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:17.271021 7509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:17.277901 7509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:17.277990 7509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:17.278100 7509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:17.287652 7509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:38023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44091
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:17.288797 7509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:17.289981 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:17.302340 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:17.302465 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:17.302493 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:17.303141 7509 server_base.cc:1034] running on GCE node
I20260619 01:00:17.303678 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:17.304442 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:17.305678 7509 hybrid_clock.cc:648] HybridClock initialized: now 1781830817305633 us; error 35 us; skew 500 ppm
I20260619 01:00:17.305989 7509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:17.309074 7509 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:00:17.310456 7509 webserver.cc:460] Webserver started at http://127.0.97.2:44091/ using document root <none> and password file <none>
I20260619 01:00:17.311305 7509 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:17.311448 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:17.316632 7509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 01:00:17.319250 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.320515 7509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:17.320684 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "46c75eef91c08f6a6c304ef0898e3c8d"
server_key_iv: "49e7e96715b6f008dd72c09024e4f4c2"
server_key_version: "encryptionkey@0"
I20260619 01:00:17.321614 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:17.343235 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:17.347445 7509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:17.347708 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:17.348623 7509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:17.349942 7509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:17.350059 7509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.350210 7509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:17.350299 7509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.352809 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: true } } }
I20260619 01:00:17.353224 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:17.355075 7533 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: attempting to promote NON_VOTER d2bbf30bacc44d58998cc8b53bcf05b4 to VOTER
I20260619 01:00:17.355976 7533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } }
I20260619 01:00:17.361223 7137 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Refusing update from remote peer 82345d268ed64aa3957b1563bd6f8d75: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 01:00:17.361281 7327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 LEARNER]: Refusing update from remote peer 82345d268ed64aa3957b1563bd6f8d75: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 01:00:17.362010 7533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 01:00:17.362432 7534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 01:00:17.365725 7532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.366730 7137 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.367704 7327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.369788 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.370039 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:17.370253 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.370489 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:17.371601 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.371860 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:17.373811 7571 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.374107 7571 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:17.374540 7570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82345d268ed64aa3957b1563bd6f8d75" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:17.374946 7570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:17.396051 7509 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38023
I20260619 01:00:17.396148 7646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38023 every 8 connection(s)
I20260619 01:00:17.397730 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:17.406448 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 7509
I20260619 01:00:17.406776 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6859
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:17.449146 7652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.401768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60425 (local address 127.0.97.61:36749)
0619 01:00:17.426338 (+ 24570us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:17.426346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:17.426368 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:00:17.427562 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:17.427570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:17.427660 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:17.427760 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:17.431478 (+ 3718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432608 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.435525 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436180 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.441770 (+ 5498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:17.441777 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:17.441781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:17.441825 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:17.445374 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.446327 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.446333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.446338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.446501 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447057 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.447062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.447065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.447426 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:17.447558 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:17.448540 (+ 982us) server_negotiation.cc:299] Negotiation successful
0619 01:00:17.448766 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24445,"thread_start_us":25077,"threads_started":1}
I20260619 01:00:17.449544 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.407869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.0.97.3:46017)
0619 01:00:17.425093 (+ 17224us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:17.425102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:17.425130 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:00:17.425198 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:17.425206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:17.425467 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:17.425645 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:17.428923 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.430350 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.432243 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.433123 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.433248 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.440805 (+ 7557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:17.440841 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:17.440861 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:17.440903 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:17.446408 (+ 5505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447258 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.447266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.447270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.447415 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447851 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.447856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.447859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.448277 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:17.448553 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:17.448889 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 01:00:17.449146 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17048,"thread_start_us":16914,"threads_started":1}
I20260619 01:00:17.450698 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.402069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37951 (local address 127.0.97.62:46621)
0619 01:00:17.423931 (+ 21862us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:17.423935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:17.424603 (+ 668us) server_negotiation.cc:407] Connection header received
0619 01:00:17.427703 (+ 3100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:17.427709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:17.427795 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:17.427889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:17.431479 (+ 3590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432638 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.435525 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436159 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436258 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.442344 (+ 6086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:17.442352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:17.442357 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:17.442397 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:17.446703 (+ 4306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447717 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.447722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.447726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.447892 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.448312 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.448320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.448326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.448686 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:17.448893 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:17.450271 (+ 1378us) server_negotiation.cc:299] Negotiation successful
0619 01:00:17.450448 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21743,"thread_start_us":21636,"threads_started":1}
I20260619 01:00:17.451162 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.426203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39997 (local address 127.0.97.60:38947)
0619 01:00:17.426847 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:17.426855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:17.427767 (+ 912us) server_negotiation.cc:407] Connection header received
0619 01:00:17.427824 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:17.427828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:17.427912 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:17.428129 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:17.431469 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432547 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.435595 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436280 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436412 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.442445 (+ 6033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:17.442454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:17.442460 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:17.442498 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:17.446335 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447123 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.447128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.447132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.447290 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:17.448104 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:17.448109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:17.448112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:17.449105 (+ 993us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:17.449239 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:17.449948 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 01:00:17.450810 (+ 862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":561,"threads_started":1}
I20260619 01:00:17.451439 7653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.401529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.97.2:60425)
0619 01:00:17.425686 (+ 24157us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:17.425704 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:17.427211 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:17.428004 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:17.428017 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:17.430156 (+ 2139us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:17.431261 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.431274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432807 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.432811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.435367 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.435380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436607 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.436968 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.437960 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:17.437986 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:17.441219 (+ 3233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:17.445572 (+ 4353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.445581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.445598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.446189 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.446650 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.446658 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.446663 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.446920 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.447648 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:17.447655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:17.448093 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 01:00:17.449369 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0619 01:00:17.449970 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23650,"mutex_wait_us":508,"spinlock_wait_cycles":1408,"thread_start_us":23432,"threads_started":1}
I20260619 01:00:17.451756 7651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.401529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:37951)
0619 01:00:17.424444 (+ 22915us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:17.424484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:17.427586 (+ 3102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:17.428340 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:17.428345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:17.430156 (+ 1811us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:17.431251 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.431275 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432800 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.432805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.435367 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.435380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436607 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.436991 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.437957 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:17.437993 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:17.442160 (+ 4167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:17.447151 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.447157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.447616 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.448023 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.448027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.448031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.448201 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.449012 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:17.449017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:17.449534 (+ 517us) client_negotiation.cc:769] Sending connection context
0619 01:00:17.450163 (+ 629us) client_negotiation.cc:240] Negotiation successful
0619 01:00:17.450452 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22618,"mutex_wait_us":392,"spinlock_wait_cycles":61696,"thread_start_us":22419,"threads_started":1}
I20260619 01:00:17.452464 6982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:17.452677 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.427000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.97.2:39997)
0619 01:00:17.427541 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:17.427557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:17.427660 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:17.428154 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:17.428158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:17.430163 (+ 2005us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:17.431247 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.431276 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.432710 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.432716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.435387 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:17.435407 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:17.436726 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:17.436733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:17.437470 (+ 737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:17.438177 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:17.438203 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:17.442233 (+ 4030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:17.446495 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.446501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.446505 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.447002 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.447422 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:17.447427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:17.447431 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:17.447610 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:17.449296 (+ 1686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:17.449301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:17.449435 (+ 134us) client_negotiation.cc:769] Sending connection context
0619 01:00:17.449605 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 01:00:17.452197 (+ 2592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":2732,"spinlock_wait_cycles":30208,"thread_start_us":290,"threads_started":1}
I20260619 01:00:17.453059 7649 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36749
I20260619 01:00:17.453459 7649 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:17.454450 7649 heartbeater.cc:511] Master 127.0.97.61:36749 requested a full tablet report, sending...
I20260619 01:00:17.455811 7648 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:17.456328 7648 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:17.456817 7648 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:17.457393 7647 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38947
I20260619 01:00:17.457513 7647 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:17.457861 7647 heartbeater.cc:511] Master 127.0.97.60:38947 requested a full tablet report, sending...
I20260619 01:00:17.458024 6530 ts_manager.cc:194] Re-registered known tserver with Master: b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:17.458009 7128 ts_manager.cc:194] Registered new tserver with Master: b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:17.459111 7317 ts_manager.cc:194] Registered new tserver with Master: b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:17.459132 6530 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37951
I20260619 01:00:17.540999 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6859
I20260619 01:00:17.557190 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46017
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46621
--webserver_port=34489
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:00:17.843401 7498 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
W20260619 01:00:17.876544 7660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:17.876839 7660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:17.876933 7660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:17.884325 7660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:17.884434 7660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:17.884549 7660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:17.894121 7660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34489
--tserver_master_addrs=127.0.97.62:46621,127.0.97.61:36749,127.0.97.60:38947
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:17.895455 7660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:17.896643 7660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:17.909214 7665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:17.909101 7668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:17.909139 7666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:17.909894 7660 server_base.cc:1034] running on GCE node
I20260619 01:00:17.910475 7660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:17.911396 7660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:17.912632 7660 hybrid_clock.cc:648] HybridClock initialized: now 1781830817912575 us; error 62 us; skew 500 ppm
I20260619 01:00:17.912940 7660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:17 dist-test-slave-3rxd krb5kdc[6486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:17.920034 7660 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:00:17.921396 7660 webserver.cc:460] Webserver started at http://127.0.97.3:34489/ using document root <none> and password file <none>
I20260619 01:00:17.922072 7660 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:17.922230 7660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:17.927814 7660 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:00:17.930608 7675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.931996 7660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:17.932168 7660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "feebc3f3881547989c6c32faf22115c5"
format_stamp: "Formatted at 2026-06-19 01:00:14 on dist-test-slave-3rxd"
server_key: "22890aa49fa91d8d4e6eb595e5e7e7f4"
server_key_iv: "40e6e09089d6dbee5155a0af4bbc0bb6"
server_key_version: "encryptionkey@0"
I20260619 01:00:17.933118 7660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:17.948894 7660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:17.953470 7660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:17.953742 7660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:17.954789 7660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:17.956038 7660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:17.956176 7660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.956298 7660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:17.956384 7660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:17.988354 7660 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46017
I20260619 01:00:17.988437 7787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46017 every 8 connection(s)
I20260619 01:00:17.989766 7660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:17.995944 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 7660
Jun 19 01:00:18 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Jun 19 01:00:18 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260619 01:00:18.044970 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.007662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57595 (local address 127.0.97.60:38947)
0619 01:00:18.021033 (+ 13371us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.021043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.021068 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:00:18.021144 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.021151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.021253 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.023342 (+ 2089us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.025269 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.027273 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.031047 (+ 3774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.031711 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.031849 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.037400 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.037407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.037413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.037452 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.041438 (+ 3986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.042332 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.042340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.042344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.042504 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.043017 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.043023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.043028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.043391 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.043575 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.044398 (+ 823us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.044609 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13205,"thread_start_us":13713,"threads_started":1}
I20260619 01:00:18.046168 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.007405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.97.3:57595)
0619 01:00:18.019463 (+ 12058us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.019482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.019589 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.021574 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.021581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.022354 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:18.023847 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.023863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.027488 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.027497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.030890 (+ 3393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.030906 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.032146 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.032151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.032393 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.033192 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:18.033243 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:18.036772 (+ 3529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:18.041575 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.041585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.041602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.042222 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.042641 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.042647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.042651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.042907 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.043545 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:18.043552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:18.043960 (+ 408us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.044294 (+ 334us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.044773 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11936,"mutex_wait_us":1266,"thread_start_us":11805,"threads_started":1}
I20260619 01:00:18.048048 7788 heartbeater.cc:347] Connected to a master server at 127.0.97.60:38947
I20260619 01:00:18.048389 7788 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:18.049316 7788 heartbeater.cc:511] Master 127.0.97.60:38947 requested a full tablet report, sending...
I20260619 01:00:18.051442 7317 ts_manager.cc:194] Registered new tserver with Master: feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017)
I20260619 01:00:18.053053 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.993229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.97.3:45933)
0619 01:00:18.015352 (+ 22123us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.015375 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.015501 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.020767 (+ 5266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.020785 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.021988 (+ 1203us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:18.023160 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.023185 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.027024 (+ 3839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.027032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.031362 (+ 4330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.031374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.032698 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.032706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.033643 (+ 937us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.035075 (+ 1432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:18.035101 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:18.041489 (+ 6388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:18.046105 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.046113 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.046121 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.046985 (+ 864us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.049982 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.049987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.049991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.050189 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.051243 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:18.051249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:18.051392 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.052197 (+ 805us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.052522 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21933,"mutex_wait_us":2174,"spinlock_wait_cycles":54912,"thread_start_us":12735,"threads_started":1}
Jun 19 01:00:18 dist-test-slave-3rxd krb5kdc[6486](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830817, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:18.057142 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.992541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45933 (local address 127.0.97.61:36749)
0619 01:00:18.019971 (+ 27430us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.019979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.020005 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:00:18.020087 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.020095 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.020205 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.024058 (+ 3853us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.024277 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.026832 (+ 2555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.031531 (+ 4699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.032283 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.032436 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.041745 (+ 9309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.041753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.041760 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.041805 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.045968 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.047478 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.047483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.047486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.047664 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.050689 (+ 3025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.050694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.050697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.051045 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.051172 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.056468 (+ 5296us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.056710 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27303,"thread_start_us":27348,"threads_started":1}
I20260619 01:00:18.067603 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.035195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.0.97.62:46621)
0619 01:00:18.035641 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.035646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.036949 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 01:00:18.037011 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.037017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.037109 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.037210 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.046589 (+ 9379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.047688 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.050638 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.051277 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.051384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.058539 (+ 7155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.058545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.058549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.058591 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.062172 (+ 3581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.064423 (+ 2251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.064428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.064433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.064600 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.065038 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.065044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.065050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.065693 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.065911 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.066925 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.067171 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":214,"threads_started":1}
I20260619 01:00:18.069908 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.993229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:43943)
0619 01:00:18.005556 (+ 12327us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.005620 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.010101 (+ 4481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.029266 (+ 19165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.029274 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.029866 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:18.030247 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.030262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.037234 (+ 6972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.037242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.039794 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.039810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.052898 (+ 13088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.052907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.053181 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.053816 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:18.053846 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:18.056736 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:18.063754 (+ 7018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.063762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.063767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.064339 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.066208 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:18.066215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:18.066222 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:18.066421 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:18.068609 (+ 2188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:18.068616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:18.068815 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.068982 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.069332 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11957,"mutex_wait_us":1,"thread_start_us":21646,"threads_started":1}
I20260619 01:00:18.070958 7790 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36749
I20260619 01:00:18.071112 7790 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:18.071542 7790 heartbeater.cc:511] Master 127.0.97.61:36749 requested a full tablet report, sending...
I20260619 01:00:18.072620 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:17.992526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43943 (local address 127.0.97.62:46621)
0619 01:00:18.028783 (+ 36257us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.028792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.028816 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:00:18.028904 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.028911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.029013 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.034058 (+ 5045us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.034226 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.037058 (+ 2832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.039989 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.046325 (+ 6336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.055517 (+ 9192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.057075 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.057082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.057087 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.057152 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.063617 (+ 6465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.065961 (+ 2344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.065967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.065972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.066089 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.068149 (+ 2060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.068153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.068156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.068474 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.068610 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.069807 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.072252 (+ 2445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36101,"thread_start_us":36040,"threads_started":1}
I20260619 01:00:18.073334 7789 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46621
I20260619 01:00:18.073455 7789 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:18.073859 7789 heartbeater.cc:511] Master 127.0.97.62:46621 requested a full tablet report, sending...
I20260619 01:00:18.073918 7128 ts_manager.cc:194] Registered new tserver with Master: feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017)
I20260619 01:00:18.075989 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.056359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49402 (local address 127.0.97.60:38947)
0619 01:00:18.056514 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.056521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.057352 (+ 831us) server_negotiation.cc:407] Connection header received
0619 01:00:18.057575 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.057581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.057675 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.057743 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.060299 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.061479 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.063617 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.064267 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.064373 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.068192 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.068198 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.068205 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.068244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.071304 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.073269 (+ 1965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.073275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.073280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.073447 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.073885 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.073893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.073897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.074260 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.074797 (+ 537us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.075455 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.075661 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:00:18.077970 6531 ts_manager.cc:194] Re-registered known tserver with Master: feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017)
I20260619 01:00:18.079206 6531 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43943
I20260619 01:00:18.085305 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.036834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.0.97.61:36749)
0619 01:00:18.057694 (+ 20860us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.057701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.060600 (+ 2899us) server_negotiation.cc:407] Connection header received
0619 01:00:18.060666 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.060673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.060772 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.060876 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.063191 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.065221 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.070634 (+ 5413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.071406 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.071506 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.076641 (+ 5135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.076650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.076657 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.076697 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.081655 (+ 4958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.083490 (+ 1835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.083495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.083499 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.083628 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.084066 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.084072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.084078 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.084371 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.084570 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.084906 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.085053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20731,"thread_start_us":21455,"threads_started":1}
I20260619 01:00:18.075129 6530 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:18.088907 6530 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:18.123793 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.107025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46017 (local address 127.0.0.1:57744)
0619 01:00:18.113948 (+ 6923us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.113972 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.114140 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.114975 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.114982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.115012 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:18.115426 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.115439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.117826 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.117835 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.120597 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.120610 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.122742 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.122749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.122939 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.122987 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.123092 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.123223 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6771,"thread_start_us":6659,"threads_started":1}
I20260619 01:00:18.124336 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.107526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.0.97.3:46017)
0619 01:00:18.108277 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.108290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.114114 (+ 5824us) server_negotiation.cc:407] Connection header received
0619 01:00:18.114364 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.114377 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.114764 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.115008 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:18.115644 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.117671 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.120752 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.122511 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.122692 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.122996 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.123192 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.123892 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":305,"threads_started":1}
I20260619 01:00:18.129136 7723 tablet_service.cc:1467] Processing CreateTablet for tablet 17522d852dd842499f6dae01794725cc (DEFAULT_TABLE table=test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:18.131064 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17522d852dd842499f6dae01794725cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:18.131842 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.106362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37835 (local address 127.0.0.1:60796)
0619 01:00:18.106927 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.106943 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.107093 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.123856 (+ 16763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.123862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.123894 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:18.125244 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.125256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.127550 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.127555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.129192 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.129203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.131108 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.131115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.131284 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.131318 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.131382 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.131499 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":275,"threads_started":1}
I20260619 01:00:18.132910 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.106675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.0.97.1:37835)
0619 01:00:18.121806 (+ 15131us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.121819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.121858 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:00:18.121980 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.121993 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.123600 (+ 1607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.125455 (+ 1855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:18.125682 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.127389 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.129388 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.130885 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.132201 (+ 1316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.132399 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.132468 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.132625 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14893,"thread_start_us":15604,"threads_started":1}
I20260619 01:00:18.136520 7432 tablet_service.cc:1467] Processing CreateTablet for tablet 17522d852dd842499f6dae01794725cc (DEFAULT_TABLE table=test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:18.137907 7432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17522d852dd842499f6dae01794725cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:18.138649 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.107225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.0.1:38354)
0619 01:00:18.126731 (+ 19506us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.126750 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.126836 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.131885 (+ 5049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.131891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.131917 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:18.132416 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.132429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.134133 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.134141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.136299 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.136331 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.137911 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.137917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.138095 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.138131 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.138249 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.138381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19391,"thread_start_us":20359,"threads_started":1}
I20260619 01:00:18.138688 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.107680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.0.97.2:38023)
0619 01:00:18.131120 (+ 23440us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.131128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.131154 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:00:18.131252 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.131260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.131487 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.131684 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:18.132594 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.133995 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.136445 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.137674 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.137899 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.138089 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.138286 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.138431 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23284,"thread_start_us":23169,"threads_started":1}
I20260619 01:00:18.145195 7582 tablet_service.cc:1467] Processing CreateTablet for tablet 17522d852dd842499f6dae01794725cc (DEFAULT_TABLE table=test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:18.146270 7820 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Bootstrap starting.
I20260619 01:00:18.146457 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17522d852dd842499f6dae01794725cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:18.150995 7820 tablet_bootstrap.cc:654] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:18.152174 7820 log.cc:824] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:18.152699 7821 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Bootstrap starting.
I20260619 01:00:18.155784 7821 tablet_bootstrap.cc:654] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:18.157621 7821 log.cc:824] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:18.157920 7820 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: No bootstrap required, opened a new log
I20260619 01:00:18.158188 7820 ts_tablet_manager.cc:1397] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260619 01:00:18.163007 7824 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Bootstrap starting.
I20260619 01:00:18.166405 7824 tablet_bootstrap.cc:654] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:18.167099 7821 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: No bootstrap required, opened a new log
I20260619 01:00:18.167692 7824 log.cc:824] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:18.167868 7820 raft_consensus.cc:348] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.168321 7820 raft_consensus.cc:374] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:18.168330 7821 ts_tablet_manager.cc:1397] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260619 01:00:18.168429 7820 raft_consensus.cc:729] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feebc3f3881547989c6c32faf22115c5, State: Initialized, Role: FOLLOWER
I20260619 01:00:18.169071 7820 consensus_queue.cc:260] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.170140 7789 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
I20260619 01:00:18.170776 7824 tablet_bootstrap.cc:492] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: No bootstrap required, opened a new log
I20260619 01:00:18.171028 7824 ts_tablet_manager.cc:1397] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260619 01:00:18.172773 7820 ts_tablet_manager.cc:1428] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Time spent starting tablet: real 0.014s user 0.001s sys 0.013s
I20260619 01:00:18.178704 7821 raft_consensus.cc:348] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.179142 7821 raft_consensus.cc:374] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:18.179260 7821 raft_consensus.cc:729] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb803022fd5945729575948839a387d5, State: Initialized, Role: FOLLOWER
I20260619 01:00:18.179828 7821 consensus_queue.cc:260] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.181564 7821 ts_tablet_manager.cc:1428] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260619 01:00:18.184965 7824 raft_consensus.cc:348] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.185392 7824 raft_consensus.cc:374] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:18.185505 7824 raft_consensus.cc:729] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dbff9291bc4b7182e503b7ee4f1d2f, State: Initialized, Role: FOLLOWER
I20260619 01:00:18.186254 7824 consensus_queue.cc:260] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.187593 7648 heartbeater.cc:503] Master 127.0.97.62:46621 was elected leader, sending a full tablet report...
I20260619 01:00:18.188306 7824 ts_tablet_manager.cc:1428] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Time spent starting tablet: real 0.017s user 0.010s sys 0.002s
W20260619 01:00:18.242601 7791 tablet.cc:2369] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:00:18.324903 7505 tablet.cc:2369] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:18.333320 7827 raft_consensus.cc:482] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:18.333623 7827 raft_consensus.cc:504] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.335273 7827 leader_election.cc:288] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017), b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:18.342193 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.335678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45961 (local address 127.0.97.3:46017)
0619 01:00:18.335944 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.335954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.336487 (+ 533us) server_negotiation.cc:407] Connection header received
0619 01:00:18.336679 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.336684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.336759 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.336859 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:18.337735 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.338477 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.340486 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.341255 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.341396 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.341502 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.341822 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.341939 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 01:00:18.342331 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.335561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46017 (local address 127.0.97.1:45961)
0619 01:00:18.336353 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.336374 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.336540 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.336906 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.336911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.336944 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:18.337595 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.337605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.338625 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.338630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.340307 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.340318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.341462 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.341468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.341666 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.341690 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.341799 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.341920 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":61,"spinlock_wait_cycles":41088,"thread_start_us":421,"threads_started":1}
I20260619 01:00:18.342877 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.335722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52779 (local address 127.0.97.2:38023)
0619 01:00:18.335915 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.335924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.336490 (+ 566us) server_negotiation.cc:407] Connection header received
0619 01:00:18.336678 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.336684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.336759 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.336859 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:18.337633 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.338900 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.340940 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.341809 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.341950 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.342056 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.342402 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.342543 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 01:00:18.343453 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.335647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.97.1:52779)
0619 01:00:18.336389 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:18.336407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:18.336531 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:18.336906 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:18.336911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:18.336950 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:18.337462 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.337472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.339030 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.339040 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.340791 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:18.340802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.342023 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.342029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:18.342203 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.342226 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:00:18.343061 (+ 835us) client_negotiation.cc:240] Negotiation successful
0619 01:00:18.343187 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":64,"spinlock_wait_cycles":4480,"thread_start_us":452,"threads_started":1}
I20260619 01:00:18.343406 7743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17522d852dd842499f6dae01794725cc" candidate_uuid: "cb803022fd5945729575948839a387d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feebc3f3881547989c6c32faf22115c5" is_pre_election: true
I20260619 01:00:18.343968 7743 raft_consensus.cc:2454] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb803022fd5945729575948839a387d5 in term 0.
I20260619 01:00:18.344470 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17522d852dd842499f6dae01794725cc" candidate_uuid: "cb803022fd5945729575948839a387d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" is_pre_election: true
I20260619 01:00:18.344611 7385 leader_election.cc:302] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb803022fd5945729575948839a387d5, feebc3f3881547989c6c32faf22115c5; no voters:
I20260619 01:00:18.344914 7602 raft_consensus.cc:2454] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb803022fd5945729575948839a387d5 in term 0.
I20260619 01:00:18.344970 7827 raft_consensus.cc:2781] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:18.345108 7827 raft_consensus.cc:482] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:18.345202 7827 raft_consensus.cc:3037] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:18.347445 7827 raft_consensus.cc:504] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.348023 7827 leader_election.cc:288] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [CANDIDATE]: Term 1 election: Requested vote from peers feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017), b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023)
I20260619 01:00:18.348378 7743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17522d852dd842499f6dae01794725cc" candidate_uuid: "cb803022fd5945729575948839a387d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feebc3f3881547989c6c32faf22115c5"
I20260619 01:00:18.348515 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17522d852dd842499f6dae01794725cc" candidate_uuid: "cb803022fd5945729575948839a387d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f"
I20260619 01:00:18.348593 7743 raft_consensus.cc:3037] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:18.348743 7602 raft_consensus.cc:3037] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:18.352200 7743 raft_consensus.cc:2454] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb803022fd5945729575948839a387d5 in term 1.
I20260619 01:00:18.352245 7602 raft_consensus.cc:2454] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb803022fd5945729575948839a387d5 in term 1.
I20260619 01:00:18.352564 7385 leader_election.cc:302] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb803022fd5945729575948839a387d5, feebc3f3881547989c6c32faf22115c5; no voters:
I20260619 01:00:18.352869 7827 raft_consensus.cc:2781] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:18.353384 7827 raft_consensus.cc:686] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 1 LEADER]: Becoming Leader. State: Replica: cb803022fd5945729575948839a387d5, State: Running, Role: LEADER
I20260619 01:00:18.354219 7827 consensus_queue.cc:237] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:18.358902 6530 catalog_manager.cc:5495] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb803022fd5945729575948839a387d5 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "cb803022fd5945729575948839a387d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } health_report { overall_health: UNKNOWN } } }
I20260619 01:00:18.421276 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.416544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60812 (local address 127.0.97.1:37835)
0619 01:00:18.416766 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.416771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.416788 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:00:18.416948 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.416951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.417043 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.417171 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:18.417491 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.418173 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.419370 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.419757 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.419843 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.420240 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:18.420716 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:18.420820 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.420913 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.420972 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 01:00:18.422214 7650 tablet.cc:2369] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:18.429883 7743 raft_consensus.cc:1261] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Refusing update from remote peer cb803022fd5945729575948839a387d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:18.429890 7602 raft_consensus.cc:1261] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Refusing update from remote peer cb803022fd5945729575948839a387d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:18.430939 7831 consensus_queue.cc:1035] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:18.431281 7827 consensus_queue.cc:1035] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:18.442914 7835 mvcc.cc:205] Tried to move back new op lower bound from 7298379032279126016 to 7298379031982362624. Current Snapshot: MvccSnapshot[applied={T|T < 7298379032279126016}]
I20260619 01:00:18.463887 6530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:18.464179 6530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:18.469894 6530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82345d268ed64aa3957b1563bd6f8d75: Sending DeleteTablet for 3 replicas of tablet 17522d852dd842499f6dae01794725cc
I20260619 01:00:18.471132 7432 tablet_service.cc:1514] Processing DeleteTablet for tablet 17522d852dd842499f6dae01794725cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60796
I20260619 01:00:18.471164 7723 tablet_service.cc:1514] Processing DeleteTablet for tablet 17522d852dd842499f6dae01794725cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57744
I20260619 01:00:18.471675 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet 17522d852dd842499f6dae01794725cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:18 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38354
I20260619 01:00:18.483347 7845 tablet_replica.cc:330] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: stopping tablet replica
I20260619 01:00:18.483668 7846 tablet_replica.cc:330] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: stopping tablet replica
I20260619 01:00:18.484138 7846 raft_consensus.cc:2229] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:18.484548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6499
I20260619 01:00:18.484671 7846 raft_consensus.cc:2258] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:18.483180 7847 tablet_replica.cc:330] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: stopping tablet replica
I20260619 01:00:18.484148 7845 raft_consensus.cc:2229] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:18.485446 7847 raft_consensus.cc:2229] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:18.485530 6561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:18.485960 7847 raft_consensus.cc:2258] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:18.486382 7845 raft_consensus.cc:2258] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:18.488798 7846 ts_tablet_manager.cc:1905] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:18.495929 7847 ts_tablet_manager.cc:1905] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:18.501573 7845 ts_tablet_manager.cc:1905] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:18.501730 7847 ts_tablet_manager.cc:1918] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:18.501883 7847 log.cc:1196] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/17522d852dd842499f6dae01794725cc
I20260619 01:00:18.502430 7847 ts_tablet_manager.cc:1939] T 17522d852dd842499f6dae01794725cc P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting consensus metadata
I20260619 01:00:18.503314 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.476800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39992 (local address 127.0.97.61:36749)
0619 01:00:18.476954 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.476960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.481759 (+ 4799us) server_negotiation.cc:407] Connection header received
0619 01:00:18.482383 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.482389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.482467 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.482581 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.487786 (+ 5205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.488968 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.491303 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.491955 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.492099 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.495346 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.495353 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.495358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.495398 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.498844 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.499615 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.499621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.499624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.499767 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.500160 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.500165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.500168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.500506 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.500797 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.502706 (+ 1909us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.502955 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 01:00:18.505805 7846 ts_tablet_manager.cc:1918] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:18.505987 7846 log.cc:1196] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/17522d852dd842499f6dae01794725cc
I20260619 01:00:18.506469 7846 ts_tablet_manager.cc:1939] T 17522d852dd842499f6dae01794725cc P feebc3f3881547989c6c32faf22115c5: Deleting consensus metadata
I20260619 01:00:18.515913 7845 ts_tablet_manager.cc:1918] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:18.516170 7845 log.cc:1196] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/17522d852dd842499f6dae01794725cc
I20260619 01:00:18.516803 7845 ts_tablet_manager.cc:1939] T 17522d852dd842499f6dae01794725cc P cb803022fd5945729575948839a387d5: Deleting consensus metadata
I20260619 01:00:18.520175 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.481884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.0.97.60:38947)
0619 01:00:18.482056 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.482068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.502182 (+ 20114us) server_negotiation.cc:407] Connection header received
0619 01:00:18.502340 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.502346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.502426 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.502519 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.505575 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.506699 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.509394 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.510045 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.510150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.513705 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.513712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.513716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.513755 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.517181 (+ 3426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.517940 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.517945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.517948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.518089 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.518542 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.518548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.518558 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.518959 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.519232 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.519576 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.519811 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:00:18.624902 6515 catalog_manager.cc:4841] TS feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017): tablet 17522d852dd842499f6dae01794725cc (table test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]) successfully deleted
I20260619 01:00:18.625555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6499
I20260619 01:00:18.625514 6516 catalog_manager.cc:4841] TS b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023): tablet 17522d852dd842499f6dae01794725cc (table test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]) successfully deleted
I20260619 01:00:18.625540 6517 catalog_manager.cc:4841] TS cb803022fd5945729575948839a387d5 (127.0.97.1:37835): tablet 17522d852dd842499f6dae01794725cc (table test-table [id=9f52f3f4b97e413697bdff07ccaf8cb8]) successfully deleted
I20260619 01:00:18.675145 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.660508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49430 (local address 127.0.97.60:38947)
0619 01:00:18.660708 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.660715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.660737 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:00:18.660937 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.660942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.661009 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.661111 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.662560 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.663618 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.665585 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.666177 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.666280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.668751 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.668757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.668761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.668796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.672057 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.672844 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.672849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.672853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.673011 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.673467 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.673472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.673474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.673818 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.674051 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.674642 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.674858 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 01:00:18.675364 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:18.659438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.0.97.61:36749)
0619 01:00:18.659656 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:18.659663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:18.660503 (+ 840us) server_negotiation.cc:407] Connection header received
0619 01:00:18.660774 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:18.660780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:18.660854 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:18.660925 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:18.663185 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.664035 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.665357 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:18.665926 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:18.666045 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:18.669318 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:18.669324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:18.669329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:18.669368 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:18.672505 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.673287 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.673293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.673297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.673429 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:18.673852 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:18.673860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:18.673865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:18.674209 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:18.674442 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:18.674822 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 01:00:18.675035 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 01:00:19.456491 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.455700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:39881)
0619 01:00:19.456181 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.456335 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
W20260619 01:00:19.456954 7648 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.457814 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.456986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:52323)
0619 01:00:19.457510 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.457665 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":346,"threads_started":1}
W20260619 01:00:19.458159 7498 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.463279 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.458878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:52897)
0619 01:00:19.463023 (+ 4145us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.463138 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4048,"thread_start_us":3966,"threads_started":1}
W20260619 01:00:19.463608 7789 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.970625 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82345d268ed64aa3957b1563bd6f8d75)
I20260619 01:00:19.970889 7860 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82345d268ed64aa3957b1563bd6f8d75)
I20260619 01:00:19.970973 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } }
I20260619 01:00:19.971088 7860 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } }
I20260619 01:00:19.971908 7860 leader_election.cc:288] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621), 82516a363b0842a6953c69829d53cf91 (127.0.97.61:36749)
I20260619 01:00:19.972564 7859 leader_election.cc:288] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621), d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60:38947)
I20260619 01:00:19.973491 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.972655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44534)
0619 01:00:19.973145 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.973294 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"threads_started":1}
I20260619 01:00:19.973513 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.972059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44518)
0619 01:00:19.973132 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.973324 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":961,"threads_started":1}
W20260619 01:00:19.974130 7112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
W20260619 01:00:19.975153 7301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.985456 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.984850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44536)
0619 01:00:19.985027 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.985238 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:00:19.985765 7112 leader_election.cc:334] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.986032 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.973041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:38947 (local address 127.0.0.1:49442)
0619 01:00:19.973668 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.973686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:19.973816 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:19.976096 (+ 2280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:19.976104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:19.976139 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:19.976517 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:19.976532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.978617 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.978625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:19.981641 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:19.981661 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.984076 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.984082 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:19.984282 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:19.984330 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 01:00:19.984433 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:00:19.984581 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":1334,"threads_started":1}
I20260619 01:00:19.986620 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.974231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49442 (local address 127.0.97.60:38947)
0619 01:00:19.975334 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:19.975342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:19.975468 (+ 126us) server_negotiation.cc:407] Connection header received
0619 01:00:19.975553 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:19.975560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:19.975652 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:19.975777 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:19.976748 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.978423 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.981873 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.983617 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.983759 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:19.983899 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:19.985766 (+ 1867us) server_negotiation.cc:299] Negotiation successful
0619 01:00:19.986000 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":937,"thread_start_us":642,"threads_started":1}
I20260619 01:00:19.986735 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.986045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44544)
0619 01:00:19.986314 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.986496 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260619 01:00:19.986917 7301 leader_election.cc:334] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.987527 7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82516a363b0842a6953c69829d53cf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" is_pre_election: true
I20260619 01:00:19.987895 7327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82516a363b0842a6953c69829d53cf91 in term 1.
I20260619 01:00:19.988384 7115 leader_election.cc:302] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82516a363b0842a6953c69829d53cf91, d2bbf30bacc44d58998cc8b53bcf05b4; no voters: 82345d268ed64aa3957b1563bd6f8d75
I20260619 01:00:19.988771 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 01:00:19.988900 7859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82345d268ed64aa3957b1563bd6f8d75)
I20260619 01:00:19.988977 7859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:00:19.991273 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.972123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36749 (local address 127.0.0.1:40016)
0619 01:00:19.973668 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.973686 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:19.973816 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:19.985549 (+ 11733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:19.985555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:19.985586 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:19.985893 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:19.985907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.987263 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.987267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:19.989213 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:19.989223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.990419 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.990425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:19.990608 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:19.990643 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 01:00:19.990799 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 01:00:19.990935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1465,"thread_start_us":1929,"threads_started":1}
I20260619 01:00:19.991290 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.972228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.97.61:36749)
0619 01:00:19.985267 (+ 13039us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:19.985273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:19.985300 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:19.985348 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:19.985351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:19.985410 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:19.985529 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:19.986111 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.987147 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.989365 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:19.990243 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:19.990403 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:19.990512 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:19.990823 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 01:00:19.990947 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12892,"thread_start_us":13446,"threads_started":1}
I20260619 01:00:19.992053 7137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "82516a363b0842a6953c69829d53cf91" is_pre_election: true
I20260619 01:00:19.992117 7859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } }
I20260619 01:00:19.992540 7137 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d2bbf30bacc44d58998cc8b53bcf05b4 in current term 2: Already voted for candidate 82516a363b0842a6953c69829d53cf91 in this term.
I20260619 01:00:19.992924 7859 leader_election.cc:288] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 election: Requested vote from peers 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621), d2bbf30bacc44d58998cc8b53bcf05b4 (127.0.97.60:38947)
I20260619 01:00:19.993065 7304 leader_election.cc:302] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d2bbf30bacc44d58998cc8b53bcf05b4; no voters: 82345d268ed64aa3957b1563bd6f8d75, 82516a363b0842a6953c69829d53cf91
I20260619 01:00:19.993307 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.992863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44548)
0619 01:00:19.993083 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.993169 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 01:00:19.993357 7860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:00:19.993432 7327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82516a363b0842a6953c69829d53cf91" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4"
I20260619 01:00:19.994215 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:19.993855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44558)
0619 01:00:19.994018 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:19.994097 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:00:19.994393 7112 leader_election.cc:334] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
I20260619 01:00:19.996080 7860 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 01:00:19.998557 7327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82516a363b0842a6953c69829d53cf91 in term 2.
I20260619 01:00:19.999492 7115 leader_election.cc:302] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82516a363b0842a6953c69829d53cf91, d2bbf30bacc44d58998cc8b53bcf05b4; no voters: 82345d268ed64aa3957b1563bd6f8d75
I20260619 01:00:19.999784 7859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:00:20.000211 7859 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [term 2 LEADER]: Becoming Leader. State: Replica: 82516a363b0842a6953c69829d53cf91, State: Running, Role: LEADER
I20260619 01:00:20.000594 7859 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } }
I20260619 01:00:20.002420 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82516a363b0842a6953c69829d53cf91. Latest consensus state: current_term: 2 leader_uuid: "82516a363b0842a6953c69829d53cf91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:20.002854 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:20.003463 7871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:20.006910 7871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:20.007653 7871 catalog_manager.cc:1259] Loaded cluster ID: 288596af42a54b8d8ce48c50288d05f8
I20260619 01:00:20.007740 7871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:20.008882 7871 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:20.009760 7871 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Loaded TSK: 0
I20260619 01:00:20.010583 7871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:20.056053 7127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:20.063012 7127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:20.067226 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.066810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44588)
0619 01:00:20.066987 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.067091 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 01:00:20.067267 7327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [term 2 FOLLOWER]: Refusing update from remote peer 82516a363b0842a6953c69829d53cf91: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 01:00:20.068189 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.067870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:44592)
0619 01:00:20.067983 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.068054 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 01:00:20.068049 7859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 01:00:20.068363 7112 consensus_peers.cc:487] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 -> Peer 82345d268ed64aa3957b1563bd6f8d75 (127.0.97.62:46621): Couldn't send request to peer 82345d268ed64aa3957b1563bd6f8d75. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 01:00:20.070837 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82516a363b0842a6953c69829d53cf91. Latest consensus state: current_term: 2 leader_uuid: "82516a363b0842a6953c69829d53cf91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:20.071130 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:20.072518 7869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "82516a363b0842a6953c69829d53cf91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:20.072785 7869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:20.073148 7860 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "82516a363b0842a6953c69829d53cf91" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "82345d268ed64aa3957b1563bd6f8d75" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46621 } } peers { permanent_uuid: "82516a363b0842a6953c69829d53cf91" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36749 } attrs { promote: false } } peers { permanent_uuid: "d2bbf30bacc44d58998cc8b53bcf05b4" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 38947 } attrs { promote: false } } }
I20260619 01:00:20.073448 7860 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2bbf30bacc44d58998cc8b53bcf05b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:00:20.096549 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.080267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46017 (local address 127.0.0.1:57748)
0619 01:00:20.080568 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.080587 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:20.080713 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:20.092085 (+ 11372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:20.092089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:20.092117 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:20.092286 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.092294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.093273 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.093278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.094609 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.094621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.095788 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.095793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.095979 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.096002 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:00:20.096110 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 01:00:20.096243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203}
I20260619 01:00:20.096565 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.080948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.0.97.3:46017)
0619 01:00:20.091808 (+ 10860us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.091812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.091827 (+ 15us) server_negotiation.cc:407] Connection header received
0619 01:00:20.091888 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.091891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.091945 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.092061 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:20.092419 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.093135 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.094804 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.095638 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.095781 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.095892 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.096144 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.096275 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10757,"thread_start_us":10659,"threads_started":1}
I20260619 01:00:20.097415 7723 tablet_service.cc:1467] Processing CreateTablet for tablet faa100295a6d472cacffb1feae96252b (DEFAULT_TABLE table=test-table [id=9d4da44e853a401992e732dcd7b99349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:20.097904 7723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa100295a6d472cacffb1feae96252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:20.103407 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.081085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38023 (local address 127.0.0.1:38370)
0619 01:00:20.090112 (+ 9027us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.090140 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:20.090297 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:20.097388 (+ 7091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:20.097392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:20.097420 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:20.097821 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.097829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.099138 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.099144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.101188 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.101201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.102484 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.102490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.102694 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.102748 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 01:00:20.102832 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 01:00:20.102957 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8874,"mutex_wait_us":27,"thread_start_us":9264,"threads_started":1}
I20260619 01:00:20.103454 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.080846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.0.97.1:37835)
0619 01:00:20.096783 (+ 15937us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.096789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.096808 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:20.096871 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.096875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.096944 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.097182 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:20.097716 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.098691 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.101314 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.102513 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.102627 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.102751 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.103046 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.103183 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15794,"thread_start_us":16197,"threads_started":1}
I20260619 01:00:20.103439 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.080777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37835 (local address 127.0.0.1:60818)
0619 01:00:20.081004 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.081025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:20.081155 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:20.097220 (+ 16065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:20.097225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:20.097252 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:20.097531 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.097541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.098870 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.098875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.101134 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.101152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.102688 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.102693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.102890 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.102917 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:00:20.102994 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 01:00:20.103132 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":75}
I20260619 01:00:20.104395 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.081575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.0.97.2:38023)
0619 01:00:20.097110 (+ 15535us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.097116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.097134 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:00:20.097201 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.097205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.097277 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.097372 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:20.098006 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.099005 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.101347 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.102282 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.102429 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.102531 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.103948 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.104089 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15418,"thread_start_us":15323,"threads_started":1}
I20260619 01:00:20.104691 7432 tablet_service.cc:1467] Processing CreateTablet for tablet faa100295a6d472cacffb1feae96252b (DEFAULT_TABLE table=test-table [id=9d4da44e853a401992e732dcd7b99349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:20.105067 7582 tablet_service.cc:1467] Processing CreateTablet for tablet faa100295a6d472cacffb1feae96252b (DEFAULT_TABLE table=test-table [id=9d4da44e853a401992e732dcd7b99349]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:20.105242 7432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa100295a6d472cacffb1feae96252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:20.105490 7582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa100295a6d472cacffb1feae96252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:20.107224 7882 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Bootstrap starting.
I20260619 01:00:20.109022 7882 tablet_bootstrap.cc:654] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:20.111178 7882 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: No bootstrap required, opened a new log
I20260619 01:00:20.111307 7882 ts_tablet_manager.cc:1397] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 01:00:20.111635 7884 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Bootstrap starting.
I20260619 01:00:20.112210 7885 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Bootstrap starting.
I20260619 01:00:20.112082 7882 raft_consensus.cc:348] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.112299 7882 raft_consensus.cc:374] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:20.112411 7882 raft_consensus.cc:729] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feebc3f3881547989c6c32faf22115c5, State: Initialized, Role: FOLLOWER
I20260619 01:00:20.112677 7882 consensus_queue.cc:260] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.113502 7882 ts_tablet_manager.cc:1428] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 01:00:20.114305 7885 tablet_bootstrap.cc:654] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:20.115033 7884 tablet_bootstrap.cc:654] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:20.121663 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.113726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:54025)
0619 01:00:20.121239 (+ 7513us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.121371 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7358,"thread_start_us":7216,"threads_started":1}
I20260619 01:00:20.122941 7885 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: No bootstrap required, opened a new log
I20260619 01:00:20.123095 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.122437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:56151)
0619 01:00:20.122867 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.122957 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328}
I20260619 01:00:20.123091 7885 ts_tablet_manager.cc:1397] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260619 01:00:20.123420 7884 tablet_bootstrap.cc:492] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: No bootstrap required, opened a new log
I20260619 01:00:20.123566 7884 ts_tablet_manager.cc:1397] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260619 01:00:20.123782 7885 raft_consensus.cc:348] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.123991 7885 raft_consensus.cc:374] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:20.124056 7885 raft_consensus.cc:729] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1dbff9291bc4b7182e503b7ee4f1d2f, State: Initialized, Role: FOLLOWER
I20260619 01:00:20.124225 7885 consensus_queue.cc:260] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.124289 7884 raft_consensus.cc:348] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.124482 7884 raft_consensus.cc:374] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:20.124586 7884 raft_consensus.cc:729] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb803022fd5945729575948839a387d5, State: Initialized, Role: FOLLOWER
I20260619 01:00:20.124792 7884 consensus_queue.cc:260] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.125555 7884 ts_tablet_manager.cc:1428] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:20.126135 7885 ts_tablet_manager.cc:1428] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:20.128018 7499 heartbeater.cc:503] Master 127.0.97.61:36749 was elected leader, sending a full tablet report...
I20260619 01:00:20.132863 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.126222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:52399)
0619 01:00:20.132477 (+ 6255us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.132603 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6112,"thread_start_us":6003,"threads_started":1}
I20260619 01:00:20.134878 7649 heartbeater.cc:503] Master 127.0.97.61:36749 was elected leader, sending a full tablet report...
I20260619 01:00:20.135353 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.134791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:56749)
0619 01:00:20.135052 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.135178 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 01:00:20.139622 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.126705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:35927)
0619 01:00:20.139239 (+ 12534us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.139342 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12381,"thread_start_us":12247,"threads_started":1}
I20260619 01:00:20.140404 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.140101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:56603)
0619 01:00:20.140248 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.140314 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:00:20.324404 7890 raft_consensus.cc:482] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:20.324618 7890 raft_consensus.cc:504] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.326335 7890 leader_election.cc:288] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017), cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:20.332863 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.326750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59909 (local address 127.0.97.1:37835)
0619 01:00:20.327027 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.327032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.327055 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:00:20.327159 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.327164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.327270 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.327377 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:20.328505 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.329378 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.331174 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.331921 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.332081 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.332187 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.332459 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.332588 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260619 01:00:20.332901 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.326671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37835 (local address 127.0.97.2:59909)
0619 01:00:20.326841 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.326860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:20.327001 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:20.327934 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:20.327939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:20.327968 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:20.328360 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.328374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.329521 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.329527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.331036 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.331047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.332104 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.332110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.332301 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.332325 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:00:20.332437 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:00:20.332589 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":51}
I20260619 01:00:20.332988 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.326787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45387 (local address 127.0.97.3:46017)
0619 01:00:20.326977 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.326983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.327766 (+ 783us) server_negotiation.cc:407] Connection header received
0619 01:00:20.328026 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.328033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.328107 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.328205 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:20.328875 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.329612 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.331320 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.332055 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.332203 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.332308 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.332611 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.332733 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:20.333416 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.326671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46017 (local address 127.0.97.2:45387)
0619 01:00:20.327664 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.327677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:20.327818 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:20.328223 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:20.328227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:20.328255 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:20.328682 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.328692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.329746 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.329753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.331170 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:20.331181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.332271 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.332277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:20.332450 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.332478 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 01:00:20.332580 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:00:20.332701 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":443,"thread_start_us":276,"threads_started":1}
I20260619 01:00:20.333714 7452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa100295a6d472cacffb1feae96252b" candidate_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb803022fd5945729575948839a387d5" is_pre_election: true
I20260619 01:00:20.334056 7452 raft_consensus.cc:2454] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dbff9291bc4b7182e503b7ee4f1d2f in term 0.
I20260619 01:00:20.334041 7743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa100295a6d472cacffb1feae96252b" candidate_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feebc3f3881547989c6c32faf22115c5" is_pre_election: true
I20260619 01:00:20.334259 7743 raft_consensus.cc:2454] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1dbff9291bc4b7182e503b7ee4f1d2f in term 0.
I20260619 01:00:20.334663 7527 leader_election.cc:302] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1dbff9291bc4b7182e503b7ee4f1d2f, cb803022fd5945729575948839a387d5; no voters:
I20260619 01:00:20.335175 7890 raft_consensus.cc:2781] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:20.335307 7890 raft_consensus.cc:482] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:20.335414 7890 raft_consensus.cc:3037] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:20.337730 7890 raft_consensus.cc:504] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.338318 7890 leader_election.cc:288] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [CANDIDATE]: Term 1 election: Requested vote from peers feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017), cb803022fd5945729575948839a387d5 (127.0.97.1:37835)
I20260619 01:00:20.338696 7743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa100295a6d472cacffb1feae96252b" candidate_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feebc3f3881547989c6c32faf22115c5"
I20260619 01:00:20.338951 7743 raft_consensus.cc:3037] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:20.338908 7452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa100295a6d472cacffb1feae96252b" candidate_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb803022fd5945729575948839a387d5"
I20260619 01:00:20.339120 7452 raft_consensus.cc:3037] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:20.341149 7743 raft_consensus.cc:2454] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dbff9291bc4b7182e503b7ee4f1d2f in term 1.
I20260619 01:00:20.341535 7525 leader_election.cc:302] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1dbff9291bc4b7182e503b7ee4f1d2f, feebc3f3881547989c6c32faf22115c5; no voters:
I20260619 01:00:20.341784 7452 raft_consensus.cc:2454] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1dbff9291bc4b7182e503b7ee4f1d2f in term 1.
I20260619 01:00:20.341812 7890 raft_consensus.cc:2781] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:20.342427 7890 raft_consensus.cc:686] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 LEADER]: Becoming Leader. State: Replica: b1dbff9291bc4b7182e503b7ee4f1d2f, State: Running, Role: LEADER
I20260619 01:00:20.342866 7890 consensus_queue.cc:237] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } }
I20260619 01:00:20.343372 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.342972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:58725)
0619 01:00:20.343144 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.343256 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 01:00:20.346522 7128 catalog_manager.cc:5495] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2). New cstate: current_term: 1 leader_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1dbff9291bc4b7182e503b7ee4f1d2f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38023 } health_report { overall_health: HEALTHY } } }
I20260619 01:00:20.396149 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.391145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.0.97.2:38023)
0619 01:00:20.391335 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.391340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.391358 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:00:20.391515 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.391519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.391607 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.391730 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:20.392111 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.392783 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.394043 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.394439 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.394528 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.394992 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:20.395500 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:20.395623 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.395733 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.395850 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:00:20.407442 7743 raft_consensus.cc:1261] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Refusing update from remote peer b1dbff9291bc4b7182e503b7ee4f1d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:20.407840 7452 raft_consensus.cc:1261] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Refusing update from remote peer b1dbff9291bc4b7182e503b7ee4f1d2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:20.408144 7895 consensus_queue.cc:1035] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "feebc3f3881547989c6c32faf22115c5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:20.408531 7890 consensus_queue.cc:1035] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb803022fd5945729575948839a387d5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:20.410295 7790 heartbeater.cc:503] Master 127.0.97.61:36749 was elected leader, sending a full tablet report...
I20260619 01:00:20.411307 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.410811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:55013)
0619 01:00:20.411018 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.411142 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 01:00:20.411867 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.411188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:48053)
0619 01:00:20.411542 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.411665 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260619 01:00:20.415336 7898 mvcc.cc:205] Tried to move back new op lower bound from 7298379040381829120 to 7298379040128405504. Current Snapshot: MvccSnapshot[applied={T|T < 7298379040381829120}]
I20260619 01:00:20.417121 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.416712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.3:42789)
0619 01:00:20.416857 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.416946 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 01:00:20.417589 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.416712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.1:36157)
0619 01:00:20.417330 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.417430 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527}
I20260619 01:00:20.418476 7899 mvcc.cc:205] Tried to move back new op lower bound from 7298379040381829120 to 7298379040128405504. Current Snapshot: MvccSnapshot[applied={T|T < 7298379040381829120}]
I20260619 01:00:20.420460 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.419966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.97.2:44899)
0619 01:00:20.420209 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.420302 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260619 01:00:20.430574 7128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:20.430859 7128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:20.436609 7128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82516a363b0842a6953c69829d53cf91: Sending DeleteTablet for 3 replicas of tablet faa100295a6d472cacffb1feae96252b
I20260619 01:00:20.437713 7723 tablet_service.cc:1514] Processing DeleteTablet for tablet faa100295a6d472cacffb1feae96252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:57748
I20260619 01:00:20.438016 7582 tablet_service.cc:1514] Processing DeleteTablet for tablet faa100295a6d472cacffb1feae96252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:38370
I20260619 01:00:20.438150 7432 tablet_service.cc:1514] Processing DeleteTablet for tablet faa100295a6d472cacffb1feae96252b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:20 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:60818
I20260619 01:00:20.440832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 7289
I20260619 01:00:20.453240 7909 tablet_replica.cc:330] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: stopping tablet replica
I20260619 01:00:20.453509 7909 raft_consensus.cc:2229] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:20.453893 7909 raft_consensus.cc:2258] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:20.456071 7908 tablet_replica.cc:330] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: stopping tablet replica
I20260619 01:00:20.456265 7908 raft_consensus.cc:2229] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:20.456858 7908 raft_consensus.cc:2258] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:20.457026 7909 ts_tablet_manager.cc:1905] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:20.458024 7908 ts_tablet_manager.cc:1905] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:20.460800 7907 tablet_replica.cc:330] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: stopping tablet replica
I20260619 01:00:20.461000 7907 raft_consensus.cc:2229] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:20.461275 7907 raft_consensus.cc:2258] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:20.465720 7908 ts_tablet_manager.cc:1918] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:20.465862 7908 log.cc:1196] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/faa100295a6d472cacffb1feae96252b
I20260619 01:00:20.466361 7908 ts_tablet_manager.cc:1939] T faa100295a6d472cacffb1feae96252b P b1dbff9291bc4b7182e503b7ee4f1d2f: Deleting consensus metadata
I20260619 01:00:20.467585 7113 catalog_manager.cc:4841] TS b1dbff9291bc4b7182e503b7ee4f1d2f (127.0.97.2:38023): tablet faa100295a6d472cacffb1feae96252b (table test-table [id=9d4da44e853a401992e732dcd7b99349]) successfully deleted
I20260619 01:00:20.468493 7909 ts_tablet_manager.cc:1918] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:20.468686 7909 log.cc:1196] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/faa100295a6d472cacffb1feae96252b
I20260619 01:00:20.468786 7907 ts_tablet_manager.cc:1905] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:20.469233 7909 ts_tablet_manager.cc:1939] T faa100295a6d472cacffb1feae96252b P cb803022fd5945729575948839a387d5: Deleting consensus metadata
I20260619 01:00:20.470314 7114 catalog_manager.cc:4841] TS cb803022fd5945729575948839a387d5 (127.0.97.1:37835): tablet faa100295a6d472cacffb1feae96252b (table test-table [id=9d4da44e853a401992e732dcd7b99349]) successfully deleted
I20260619 01:00:20.473519 7907 ts_tablet_manager.cc:1918] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:20.473660 7907 log.cc:1196] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/faa100295a6d472cacffb1feae96252b
I20260619 01:00:20.474032 7907 ts_tablet_manager.cc:1939] T faa100295a6d472cacffb1feae96252b P feebc3f3881547989c6c32faf22115c5: Deleting consensus metadata
I20260619 01:00:20.474938 7112 catalog_manager.cc:4841] TS feebc3f3881547989c6c32faf22115c5 (127.0.97.3:46017): tablet faa100295a6d472cacffb1feae96252b (table test-table [id=9d4da44e853a401992e732dcd7b99349]) successfully deleted
I20260619 01:00:20.478454 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.442241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38068 (local address 127.0.97.1:37835)
0619 01:00:20.442451 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.442459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.461936 (+ 19477us) server_negotiation.cc:407] Connection header received
0619 01:00:20.462130 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.462136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.462217 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.462330 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:20.464314 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.465539 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.467578 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.468199 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.468294 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.471914 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:20.471932 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:20.471951 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:20.472005 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:20.475795 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.476429 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.476439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.476443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.476592 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.476929 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.476934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.476938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.477398 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:20.477695 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.477880 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.478104 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:00:20.479107 7492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:20.541672 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.541152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:49338)
0619 01:00:20.541330 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.541453 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 01:00:20.542892 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.542487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46621 (local address 127.0.0.1:49346)
0619 01:00:20.542647 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:20.542753 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 01:00:20.598661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 7289
I20260619 01:00:20.629758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 7509
I20260619 01:00:20.644090 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.630489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.0.97.2:38023)
0619 01:00:20.630675 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.630681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.630766 (+ 85us) server_negotiation.cc:407] Connection header received
0619 01:00:20.630975 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.630984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.631069 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.631187 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:20.632424 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.633278 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.634657 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.635145 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.635288 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.637933 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:20.637957 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:20.637977 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:20.638009 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:20.641306 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.642026 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.642034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.642037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.642185 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.642560 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.642566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.642569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.642955 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:20.643262 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.643541 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.643749 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:00:20.644769 7642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:20.761873 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 7509
I20260619 01:00:20.776837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 7660
I20260619 01:00:20.789657 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:20.777600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.0.97.3:46017)
0619 01:00:20.777777 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:20.777785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:20.777806 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:00:20.778040 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:20.778044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:20.778114 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:20.778219 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:20.779564 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.780298 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.781484 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:20.781891 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:20.782025 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:20.784415 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:20.784429 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:20.784446 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:20.784474 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:20.787048 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.787728 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.787735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.787738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.787864 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:20.788222 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:20.788228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:20.788231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:20.788572 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:20.788857 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:20.789151 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 01:00:20.789357 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:00:20.790341 7783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:20.882035 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 7660
I20260619 01:00:20.899660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 6992
I20260619 01:00:20.900701 7158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:21.019829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 6992
I20260619 01:00:21.040722 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 7092
I20260619 01:00:21.041713 7347 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:21.155505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 7092
2026-06-19T01:00:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9619 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:21 dist-test-slave-3rxd krb5kdc[7918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:21 dist-test-slave-3rxd krb5kdc[7918](info): set up 2 sockets
Jun 19 01:00:21 dist-test-slave-3rxd krb5kdc[7918](info): commencing operation
krb5kdc: starting...
W20260619 01:00:22.263867 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:22 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:22Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260619 01:00:22.470992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44169
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:22.786396 7931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:22.786669 7931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:22.786844 7931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:22.793887 7931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:22.793977 7931 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 01:00:22.794055 7931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:22.794148 7931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:22.794257 7931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:22.794376 7931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:22.804042 7931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:22.805246 7931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:22.806563 7931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:22.818145 7936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:22.818252 7939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:22.818192 7937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:22.819262 7931 server_base.cc:1034] running on GCE node
I20260619 01:00:22.819994 7931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:22.821249 7931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:22.822459 7931 hybrid_clock.cc:648] HybridClock initialized: now 1781830822822405 us; error 35 us; skew 500 ppm
I20260619 01:00:22.822827 7931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:22 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830822, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:22.825838 7931 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260619 01:00:22.827255 7931 webserver.cc:460] Webserver started at http://127.0.97.62:34103/ using document root <none> and password file <none>
I20260619 01:00:22.827937 7931 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:22.828022 7931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:22.828276 7931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:22.831213 7931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "4c70f922f548440e82eec8b1247a6211"
format_stamp: "Formatted at 2026-06-19 01:00:22 on dist-test-slave-3rxd"
server_key: "a25a77ee2dc6ec78971e0ac213ed0664"
server_key_iv: "abe367cca1fa98bf089fdea22e63dc17"
server_key_version: "encryptionkey@0"
I20260619 01:00:22.831943 7931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c70f922f548440e82eec8b1247a6211"
format_stamp: "Formatted at 2026-06-19 01:00:22 on dist-test-slave-3rxd"
server_key: "a25a77ee2dc6ec78971e0ac213ed0664"
server_key_iv: "abe367cca1fa98bf089fdea22e63dc17"
server_key_version: "encryptionkey@0"
I20260619 01:00:22.836794 7931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:22.840061 7946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:22.841384 7931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:22.841601 7931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "4c70f922f548440e82eec8b1247a6211"
format_stamp: "Formatted at 2026-06-19 01:00:22 on dist-test-slave-3rxd"
server_key: "a25a77ee2dc6ec78971e0ac213ed0664"
server_key_iv: "abe367cca1fa98bf089fdea22e63dc17"
server_key_version: "encryptionkey@0"
I20260619 01:00:22.841822 7931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:22.872825 7931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:22.876839 7931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:22.877122 7931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:22.898152 7931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44169
I20260619 01:00:22.898186 8007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44169 every 8 connection(s)
I20260619 01:00:22.899487 7931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:22.901018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 7931
I20260619 01:00:22.901235 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:22.901857 388 external_mini_cluster.cc:1428] Setting key 88705dc407ecc652bd3420e839c72c4e
I20260619 01:00:22.903656 8008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:00:22 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:22.916561 8008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Bootstrap starting.
I20260619 01:00:22.920624 8008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:22.921866 8008 log.cc:824] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:22.925145 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:22.903836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.0.97.62:44169)
0619 01:00:22.904605 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:22.904621 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:22.904677 (+ 56us) server_negotiation.cc:407] Connection header received
0619 01:00:22.906721 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:22.906765 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:22.907286 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:22.907703 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:22.908927 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:22.910995 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:22.912169 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:22.913176 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:22.913356 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:22.917054 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:22.917065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:22.917082 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:22.917128 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:22.920885 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:22.921511 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:22.921525 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:22.921536 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:22.921732 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:22.922144 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:22.922148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:22.922150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:22.922643 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:22.923145 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:22.923462 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 01:00:22.923868 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":311,"threads_started":1}
I20260619 01:00:22.925752 8008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: No bootstrap required, opened a new log
I20260619 01:00:22.933205 8008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } }
I20260619 01:00:22.933514 8008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:22.933593 8008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c70f922f548440e82eec8b1247a6211, State: Initialized, Role: FOLLOWER
I20260619 01:00:22.934167 8008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } }
I20260619 01:00:22.934438 8008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:22.934566 8008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:22.934780 8008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:22.937134 8008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } }
I20260619 01:00:22.937610 8008 leader_election.cc:302] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c70f922f548440e82eec8b1247a6211; no voters:
I20260619 01:00:22.938138 8008 leader_election.cc:288] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:22.938264 8013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:22.938992 8013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [term 1 LEADER]: Becoming Leader. State: Replica: 4c70f922f548440e82eec8b1247a6211, State: Running, Role: LEADER
I20260619 01:00:22.939497 8013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } }
I20260619 01:00:22.940162 8008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:22.942266 8015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c70f922f548440e82eec8b1247a6211. Latest consensus state: current_term: 1 leader_uuid: "4c70f922f548440e82eec8b1247a6211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } } }
I20260619 01:00:22.942775 8015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:22.943377 8014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c70f922f548440e82eec8b1247a6211" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c70f922f548440e82eec8b1247a6211" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44169 } } }
I20260619 01:00:22.943895 8014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:22.947907 8019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:22.952347 8019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:22.960175 8019 catalog_manager.cc:1347] Generated new cluster ID: 1576b96f713b4ba98ecd4feed00b13a8
I20260619 01:00:22.960284 8019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:22.973876 8019 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:22.975617 8019 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:22.985054 8019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Generated new TSK 0
I20260619 01:00:22.986007 8019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260619 01:00:23.042464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44169
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:23.364672 8034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:23.364980 8034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:23.365162 8034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:23.372190 8034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:23.372337 8034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:23.372498 8034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:00:23.372644 8034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:23.382040 8034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:23.383315 8034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:23.384661 8034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:23.397571 8040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:23.397650 8042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:23.397653 8039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:23.398466 8034 server_base.cc:1034] running on GCE node
I20260619 01:00:23.399094 8034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:23.399993 8034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:23.401250 8034 hybrid_clock.cc:648] HybridClock initialized: now 1781830823401171 us; error 51 us; skew 500 ppm
I20260619 01:00:23.401523 8034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:23 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:23.404578 8034 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:00:23.406081 8034 webserver.cc:460] Webserver started at http://127.0.97.1:33861/ using document root <none> and password file <none>
I20260619 01:00:23.406874 8034 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:23.406966 8034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:23.407235 8034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:23.410096 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "0d5b7badfda34f5382c6a02d481b7537"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "1d7393c2f4d9b19fd49d079ccf1cec0e"
server_key_iv: "6f22ae47ba939b79dec45ce44b8f47fe"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.410910 8034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0d5b7badfda34f5382c6a02d481b7537"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "1d7393c2f4d9b19fd49d079ccf1cec0e"
server_key_iv: "6f22ae47ba939b79dec45ce44b8f47fe"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.415946 8034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:00:23.419412 8049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:23.420717 8034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:23.420879 8034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "0d5b7badfda34f5382c6a02d481b7537"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "1d7393c2f4d9b19fd49d079ccf1cec0e"
server_key_iv: "6f22ae47ba939b79dec45ce44b8f47fe"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.421110 8034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:23.439113 8034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:23.443260 8034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:23.443523 8034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:23.446544 8034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:23.446655 8034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:23.446822 8034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:23.446893 8034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:00:23 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:23.471298 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.448180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38107 (local address 127.0.97.62:44169)
0619 01:00:23.449288 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:23.449294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:23.449317 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:00:23.451138 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:23.451144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:23.451229 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:23.451361 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:23.453408 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.454359 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.457588 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.458217 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.458363 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.463591 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:23.463602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:23.463611 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:23.463659 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:23.467447 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.468412 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.468421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.468427 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.468610 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.469125 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.469133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.469139 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.469590 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:23.469779 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:23.470468 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 01:00:23.470810 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":869,"threads_started":1}
I20260619 01:00:23.472348 8058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.447620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.1:38107)
0619 01:00:23.448544 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:23.448619 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:23.450863 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:23.451598 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:23.451610 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:23.452331 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:23.453245 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.453264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.454545 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.454553 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.457420 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.457436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.458496 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.458510 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.458942 (+ 432us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.459822 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:23.459860 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:23.463314 (+ 3454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:23.467612 (+ 4298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.467622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.467645 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.468254 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.468760 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.468764 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.468767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.468970 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.469749 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:23.469755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:23.470066 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 01:00:23.470337 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 01:00:23.470864 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":335,"threads_started":1}
I20260619 01:00:23.494299 8034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38425
I20260619 01:00:23.494329 8169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38425 every 8 connection(s)
I20260619 01:00:23.495946 8034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:23.498162 8170 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44169
I20260619 01:00:23.498380 8170 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:23.499320 8170 heartbeater.cc:511] Master 127.0.97.62:44169 requested a full tablet report, sending...
I20260619 01:00:23.499523 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.483312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.1:55551)
0619 01:00:23.484250 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:23.484271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:23.484410 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:23.485017 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:23.485022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:23.485478 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:23.485784 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.485792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.487386 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.487393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.489503 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.489529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.490906 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.490915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.491108 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.491875 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:23.491909 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:23.492964 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:23.496739 (+ 3775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.496744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.496748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.497279 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.497739 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.497744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.497747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.497900 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.498545 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:23.498549 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:23.498760 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 01:00:23.498896 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 01:00:23.499141 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":622,"threads_started":1}
I20260619 01:00:23.499593 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.483085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55551 (local address 127.0.97.62:44169)
0619 01:00:23.483257 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:23.483264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:23.484359 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 01:00:23.484560 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:23.484568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:23.484679 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:23.484811 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:23.485928 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.487158 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.489697 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.490516 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.490691 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.493172 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:23.493182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:23.493190 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:23.493234 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:23.496588 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.497405 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.497413 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.497418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.497613 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.498040 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.498047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.498053 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.498412 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:23.498554 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:23.499019 (+ 465us) server_negotiation.cc:299] Negotiation successful
0619 01:00:23.499237 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 01:00:23.501750 7963 ts_manager.cc:194] Registered new tserver with Master: 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425)
I20260619 01:00:23.503257 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38107
I20260619 01:00:23.505043 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8034
I20260619 01:00:23.505231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:23.506089 388 external_mini_cluster.cc:1428] Setting key 3759b9e8def39bb5feb72db6e536c624
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260619 01:00:23.553859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44169
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:23.868309 8174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:23.868587 8174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:23.868755 8174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:23.875855 8174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:23.875970 8174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:23.876132 8174 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:00:23.876255 8174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:23.885548 8174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:23.886765 8174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:23.888033 8174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:23.901002 8180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:23.901162 8182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:23.903010 8179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:23.903232 8174 server_base.cc:1034] running on GCE node
I20260619 01:00:23.903920 8174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:23.904932 8174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:23.906185 8174 hybrid_clock.cc:648] HybridClock initialized: now 1781830823906094 us; error 53 us; skew 500 ppm
I20260619 01:00:23.906534 8174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:23 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:23.909983 8174 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:00:23.912847 8174 webserver.cc:460] Webserver started at http://127.0.97.2:37171/ using document root <none> and password file <none>
I20260619 01:00:23.913529 8174 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:23.913661 8174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:23.913972 8174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:23.916812 8174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "1267de06ed2847b88b5a3bad45602b6a"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "b5bf4962496342580edddc229d2d2210"
server_key_iv: "a13cc2eda3ab9a830fa636c817c9b605"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.917577 8174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1267de06ed2847b88b5a3bad45602b6a"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "b5bf4962496342580edddc229d2d2210"
server_key_iv: "a13cc2eda3ab9a830fa636c817c9b605"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.922626 8174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 01:00:23.926002 8189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:23.927338 8174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:00:23.927507 8174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "1267de06ed2847b88b5a3bad45602b6a"
format_stamp: "Formatted at 2026-06-19 01:00:23 on dist-test-slave-3rxd"
server_key: "b5bf4962496342580edddc229d2d2210"
server_key_iv: "a13cc2eda3ab9a830fa636c817c9b605"
server_key_version: "encryptionkey@0"
I20260619 01:00:23.927738 8174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:23.947449 8174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:23.951220 8174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:23.951478 8174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:23.954243 8174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:23.954355 8174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:23.954507 8174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:23.954581 8174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:00:23 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830823, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:23.977689 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.954446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39571 (local address 127.0.97.62:44169)
0619 01:00:23.954655 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:23.954661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:23.956039 (+ 1378us) server_negotiation.cc:407] Connection header received
0619 01:00:23.958015 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:23.958021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:23.958111 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:23.958218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:23.960257 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.961586 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.964157 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.964604 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.964746 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.969676 (+ 4930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:23.969686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:23.969692 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:23.969750 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:23.973614 (+ 3864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.974359 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.974382 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.974388 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.974546 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.975181 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.975186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.975189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.975590 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:23.975751 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:23.976827 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0619 01:00:23.977138 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:00:23.978308 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.955008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.2:39571)
0619 01:00:23.955862 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:23.955906 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:23.957743 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:23.958436 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:23.958444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:23.959215 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:23.960024 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.960057 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.961798 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.961806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.963982 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.963998 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.964847 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.964857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.965266 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.966169 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:23.966190 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:23.969390 (+ 3200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:23.973749 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.973758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.973774 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.974201 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.974659 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.974665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.974669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.975000 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.975830 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:23.975835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:23.976354 (+ 519us) client_negotiation.cc:769] Sending connection context
0619 01:00:23.976698 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 01:00:23.977151 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":370,"threads_started":1}
I20260619 01:00:24.001319 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.985047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52711 (local address 127.0.97.62:44169)
0619 01:00:23.985210 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:23.985219 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:23.985917 (+ 698us) server_negotiation.cc:407] Connection header received
0619 01:00:23.986084 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:23.986092 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:23.986190 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:23.986325 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:23.987295 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.988548 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.991041 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.991418 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.991554 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.994286 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:23.994294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:23.994302 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:23.994343 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:23.998275 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.998972 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.998980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.998986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:23.999155 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:23.999586 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:23.999591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:23.999594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.000091 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:24.000249 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.000942 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.001111 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:24.001879 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:23.985170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.2:52711)
0619 01:00:23.985803 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:23.985822 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:23.985957 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:23.986325 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:23.986328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:23.986764 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:23.987113 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.987122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.988726 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.988735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.990906 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:23.990921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:23.991646 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:23.991656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:23.991893 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:23.992606 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:23.992641 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:23.994042 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:23.998406 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.998410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.998413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.998857 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:23.999259 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:23.999264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:23.999268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:23.999431 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:24.000348 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:24.000356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:24.000614 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.000796 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.001224 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":559,"threads_started":1}
I20260619 01:00:24.007495 8174 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38043
I20260619 01:00:24.007539 8308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38043 every 8 connection(s)
I20260619 01:00:24.008630 8174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:24.010444 8309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44169
I20260619 01:00:24.010677 8309 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:24.011291 8309 heartbeater.cc:511] Master 127.0.97.62:44169 requested a full tablet report, sending...
I20260619 01:00:24.012609 7963 ts_manager.cc:194] Registered new tserver with Master: 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043)
I20260619 01:00:24.013350 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39571
I20260619 01:00:24.013778 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8174
I20260619 01:00:24.013983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:24.014549 388 external_mini_cluster.cc:1428] Setting key 9f9563486349687224f7f608b707083a
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:00:24.062021 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44169
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:24.375850 8313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:24.376173 8313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:24.376343 8313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:24.383440 8313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:24.383586 8313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:24.383728 8313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 01:00:24.383867 8313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:24.393115 8313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:44169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:24.394268 8313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:24.395535 8313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:24.407920 8318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:24.407953 8319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:24.408116 8321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:24.408792 8313 server_base.cc:1034] running on GCE node
I20260619 01:00:24.409346 8313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:24.410224 8313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:24.411442 8313 hybrid_clock.cc:648] HybridClock initialized: now 1781830824411381 us; error 40 us; skew 500 ppm
I20260619 01:00:24.411736 8313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:24.414765 8313 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:00:24.416582 8313 webserver.cc:460] Webserver started at http://127.0.97.3:33955/ using document root <none> and password file <none>
I20260619 01:00:24.417234 8313 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:24.417366 8313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:24.417688 8313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:24.420524 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "3998af71bd8249f2a5565c082539bea1"
format_stamp: "Formatted at 2026-06-19 01:00:24 on dist-test-slave-3rxd"
server_key: "264794f4e96f46f4261dbf119b5b49e1"
server_key_iv: "cd1fe917db930cde9d039021f5ff2e89"
server_key_version: "encryptionkey@0"
I20260619 01:00:24.421306 8313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3998af71bd8249f2a5565c082539bea1"
format_stamp: "Formatted at 2026-06-19 01:00:24 on dist-test-slave-3rxd"
server_key: "264794f4e96f46f4261dbf119b5b49e1"
server_key_iv: "cd1fe917db930cde9d039021f5ff2e89"
server_key_version: "encryptionkey@0"
I20260619 01:00:24.426597 8313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:00:24.429847 8328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:24.431231 8313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:24.431398 8313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "3998af71bd8249f2a5565c082539bea1"
format_stamp: "Formatted at 2026-06-19 01:00:24 on dist-test-slave-3rxd"
server_key: "264794f4e96f46f4261dbf119b5b49e1"
server_key_iv: "cd1fe917db930cde9d039021f5ff2e89"
server_key_version: "encryptionkey@0"
I20260619 01:00:24.431619 8313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:24.454135 8313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:24.457964 8313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:24.458248 8313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:24.460842 8313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:24.460958 8313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:24.461136 8313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:24.461221 8313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830824, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:24.483213 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.461286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46711 (local address 127.0.97.62:44169)
0619 01:00:24.461481 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.461487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.462917 (+ 1430us) server_negotiation.cc:407] Connection header received
0619 01:00:24.465002 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.465010 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.465121 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.465271 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:24.467671 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.468766 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.470486 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.471128 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.471264 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.476596 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:24.476606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:24.476613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:24.476666 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:24.480210 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.480940 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.480945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.480949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.481075 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.481495 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.481500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.481503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.481795 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:24.481965 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.482637 (+ 672us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.482873 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:24.484194 8337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.461853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.3:46711)
0619 01:00:24.462737 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.462782 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.464686 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.465551 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.465566 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.466405 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:24.467390 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.467426 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.468931 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.468942 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.470335 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.470346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.471364 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.471371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.471651 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.472580 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:24.472623 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:24.476305 (+ 3682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:24.480360 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:24.480369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:24.480385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:24.480833 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:24.481183 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:24.481189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:24.481193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:24.481380 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:24.481931 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:24.481941 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:24.482272 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.482535 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.483040 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":345,"threads_started":1}
I20260619 01:00:24.505796 8170 heartbeater.cc:503] Master 127.0.97.62:44169 was elected leader, sending a full tablet report...
I20260619 01:00:24.506405 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.489894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.97.3:39141)
0619 01:00:24.490565 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.490587 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.490767 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.491428 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.491440 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.492271 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:24.492943 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.492957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.494428 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.494437 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.496699 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.496719 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.497878 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.497886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.498101 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.498873 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:24.498895 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:24.499839 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:24.502883 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:24.502886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:24.502889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:24.503489 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:24.503935 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:24.503941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:24.503945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:24.504145 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:24.504965 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:24.504973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:24.505250 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.505442 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.505868 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":390,"threads_started":1}
I20260619 01:00:24.507627 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.489861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39141 (local address 127.0.97.62:44169)
0619 01:00:24.490047 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.490054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.490673 (+ 619us) server_negotiation.cc:407] Connection header received
0619 01:00:24.490938 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.490945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.491049 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.491192 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:24.493204 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.494260 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.496875 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.497505 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.497637 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.500028 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:24.500034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:24.500040 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:24.500086 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:24.502682 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.503629 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.503634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.503638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.503803 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.504318 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.504325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.504330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.504733 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:24.504887 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.506854 (+ 1967us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.507155 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:00:24.519738 8313 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45293
I20260619 01:00:24.519775 8447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45293 every 8 connection(s)
I20260619 01:00:24.520943 8313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:24.522893 8448 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44169
I20260619 01:00:24.523066 8448 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:24.523635 8448 heartbeater.cc:511] Master 127.0.97.62:44169 requested a full tablet report, sending...
I20260619 01:00:24.524842 7963 ts_manager.cc:194] Registered new tserver with Master: 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:24.525743 7963 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46711
I20260619 01:00:24.530809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8313
I20260619 01:00:24.530967 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:24.531383 388 external_mini_cluster.cc:1428] Setting key 0c6dbedec3456cde0c37953bb17163cb
I20260619 01:00:24.533926 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:00:24.578429 8059 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:24.569014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.0.97.62:44169)
0619 01:00:24.569230 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.569239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.569373 (+ 134us) server_negotiation.cc:407] Connection header received
0619 01:00:24.569517 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.569521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.569589 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.569673 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:24.570939 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.571626 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.572788 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.573186 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.573275 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.578222 (+ 4947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55078: BlockingRecv error: recv got EOF from 127.0.0.1:55078 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 01:00:24 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:24.593647 8059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.581587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.0.97.62:44169)
0619 01:00:24.581744 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.581748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.582032 (+ 284us) server_negotiation.cc:407] Connection header received
0619 01:00:24.582152 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.582156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.582220 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.582293 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:24.583388 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.584004 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.585187 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.585564 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.585649 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.589041 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:24.589046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:24.589049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:24.589076 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:24.591656 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.592177 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.592180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.592182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.592287 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:24.592563 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:24.592566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:24.592568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:24.592796 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:24.592998 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.593172 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.593324 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:00:24.597492 7963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:24.604243 7963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:24.639004 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.615581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45293 (local address 127.0.0.1:47166)
0619 01:00:24.628855 (+ 13274us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.628878 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.629000 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.630929 (+ 1929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.630936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.630965 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:24.631369 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.631383 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633761 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.633765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.636586 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.636597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.637949 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.637955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.638142 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638192 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.638343 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.638554 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13108,"mutex_wait_us":226,"thread_start_us":12936,"threads_started":1}
I20260619 01:00:24.639164 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.616501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.0.97.1:38425)
0619 01:00:24.629540 (+ 13039us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.629550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.630788 (+ 1238us) server_negotiation.cc:407] Connection header received
0619 01:00:24.630910 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.630917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.631156 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.631376 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:24.632435 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633876 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.636428 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.637857 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.637980 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638178 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.638692 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.638891 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12859,"thread_start_us":12730,"threads_started":1}
I20260619 01:00:24.639528 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.616258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38425 (local address 127.0.0.1:52370)
0619 01:00:24.630466 (+ 14208us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.630488 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.630613 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.631603 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.631607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.631635 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:24.632215 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.632226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633999 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.634006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.636221 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.636236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.638329 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.638336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.638499 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638542 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.638647 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.638784 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14083,"mutex_wait_us":171,"thread_start_us":11443,"threads_started":1}
I20260619 01:00:24.639644 8469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.615438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.0.1:43270)
0619 01:00:24.627221 (+ 11783us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.627281 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.627498 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.631391 (+ 3893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.631395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.631419 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:24.631881 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.631893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633748 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.633756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.636761 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.636773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.638389 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.638395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.638582 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638611 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.638704 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.638843 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11458,"mutex_wait_us":332,"thread_start_us":14753,"threads_started":1}
I20260619 01:00:24.640015 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.616442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.0.97.3:45293)
0619 01:00:24.630183 (+ 13741us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.630195 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.630238 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:00:24.630356 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.630368 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.630667 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.631574 (+ 907us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:24.631906 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633613 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.636767 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.637761 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.637909 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638149 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.639534 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.639706 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13512,"thread_start_us":13298,"threads_started":1}
I20260619 01:00:24.640383 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.615639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.0.97.2:38043)
0619 01:00:24.630843 (+ 15204us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.630852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.630879 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:24.630970 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.630975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.631192 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.631398 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:24.632668 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.633610 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.636930 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.638184 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.638320 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.638493 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.639932 (+ 1439us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.640090 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14995,"thread_start_us":14864,"threads_started":1}
I20260619 01:00:24.643414 8378 tablet_service.cc:1467] Processing CreateTablet for tablet fe991af8272948a4b07467542a5e1383 (DEFAULT_TABLE table=test-table [id=83fc9f28d367458f86698935316fec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.643479 8100 tablet_service.cc:1467] Processing CreateTablet for tablet fe991af8272948a4b07467542a5e1383 (DEFAULT_TABLE table=test-table [id=83fc9f28d367458f86698935316fec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.644124 8239 tablet_service.cc:1467] Processing CreateTablet for tablet fe991af8272948a4b07467542a5e1383 (DEFAULT_TABLE table=test-table [id=83fc9f28d367458f86698935316fec38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.644713 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe991af8272948a4b07467542a5e1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.644809 8100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe991af8272948a4b07467542a5e1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.645277 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe991af8272948a4b07467542a5e1383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.655803 8472 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Bootstrap starting.
I20260619 01:00:24.659082 8473 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Bootstrap starting.
I20260619 01:00:24.659297 8472 tablet_bootstrap.cc:654] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:24.661521 8472 log.cc:824] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:24.662426 8473 tablet_bootstrap.cc:654] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:24.662920 8474 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Bootstrap starting.
I20260619 01:00:24.663599 8473 log.cc:824] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:24.666098 8474 tablet_bootstrap.cc:654] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:24.666322 8473 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: No bootstrap required, opened a new log
I20260619 01:00:24.666599 8473 ts_tablet_manager.cc:1397] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 01:00:24.667264 8474 log.cc:824] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:24.667589 8472 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: No bootstrap required, opened a new log
I20260619 01:00:24.667860 8472 ts_tablet_manager.cc:1397] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260619 01:00:24.670297 8474 tablet_bootstrap.cc:492] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: No bootstrap required, opened a new log
I20260619 01:00:24.670527 8474 ts_tablet_manager.cc:1397] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 01:00:24.674933 8472 raft_consensus.cc:348] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.675308 8472 raft_consensus.cc:374] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:24.675412 8472 raft_consensus.cc:729] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Initialized, Role: FOLLOWER
I20260619 01:00:24.675985 8472 consensus_queue.cc:260] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.676998 8448 heartbeater.cc:503] Master 127.0.97.62:44169 was elected leader, sending a full tablet report...
I20260619 01:00:24.676821 8473 raft_consensus.cc:348] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.677177 8473 raft_consensus.cc:374] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:24.677254 8473 raft_consensus.cc:729] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Initialized, Role: FOLLOWER
I20260619 01:00:24.677605 8472 ts_tablet_manager.cc:1428] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260619 01:00:24.677865 8473 consensus_queue.cc:260] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.678995 8309 heartbeater.cc:503] Master 127.0.97.62:44169 was elected leader, sending a full tablet report...
I20260619 01:00:24.681116 8474 raft_consensus.cc:348] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.681655 8474 raft_consensus.cc:374] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:24.681758 8474 raft_consensus.cc:729] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Initialized, Role: FOLLOWER
I20260619 01:00:24.682296 8474 consensus_queue.cc:260] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.683120 8473 ts_tablet_manager.cc:1428] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260619 01:00:24.683908 8474 ts_tablet_manager.cc:1428] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260619 01:00:24.750398 8171 tablet.cc:2369] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:00:24.760504 8310 tablet.cc:2369] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:00:24.772399 8449 tablet.cc:2369] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:24.869957 8480 raft_consensus.cc:482] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:24.870275 8480 raft_consensus.cc:504] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.871979 8480 leader_election.cc:288] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:24.878573 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.872370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56481 (local address 127.0.97.2:38043)
0619 01:00:24.872568 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.872575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.873173 (+ 598us) server_negotiation.cc:407] Connection header received
0619 01:00:24.873382 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.873387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.873456 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.873546 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:24.874161 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.875060 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.876909 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.877650 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.877801 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.877909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.878202 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.878321 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:24.878610 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.872462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39167 (local address 127.0.97.3:45293)
0619 01:00:24.872757 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.872765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.873377 (+ 612us) server_negotiation.cc:407] Connection header received
0619 01:00:24.873565 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.873571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.873641 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.873733 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:24.874383 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.875207 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.876947 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.877684 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.877831 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.877941 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.878240 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.878357 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260619 01:00:24.879024 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.872184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.97.1:56481)
0619 01:00:24.873051 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.873067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.873246 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.873572 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.873576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.873600 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:24.874016 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.874026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.875178 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.875185 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.876759 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.876773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.877867 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.877872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.878027 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.878065 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.878170 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.878289 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":2,"spinlock_wait_cycles":21760,"thread_start_us":531,"threads_started":1}
I20260619 01:00:24.879091 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.872430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45293 (local address 127.0.97.1:39167)
0619 01:00:24.873280 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:24.873294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:24.873436 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:24.873752 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:24.873756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:24.873787 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:24.874246 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.874255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.875357 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.875362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.876798 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:24.876809 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.877906 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.877912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:24.878092 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.878114 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:00:24.878216 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:00:24.878339 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":21,"thread_start_us":528,"threads_started":1}
I20260619 01:00:24.880074 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe991af8272948a4b07467542a5e1383" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1" is_pre_election: true
I20260619 01:00:24.880029 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe991af8272948a4b07467542a5e1383" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a" is_pre_election: true
I20260619 01:00:24.880539 8263 raft_consensus.cc:2454] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 0.
I20260619 01:00:24.880537 8403 raft_consensus.cc:2454] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 0.
I20260619 01:00:24.881178 8052 leader_election.cc:302] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 1267de06ed2847b88b5a3bad45602b6a, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:24.881537 8480 raft_consensus.cc:2781] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:24.881688 8480 raft_consensus.cc:482] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:24.881814 8480 raft_consensus.cc:3037] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:24.884003 8480 raft_consensus.cc:504] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.884533 8480 leader_election.cc:288] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 election: Requested vote from peers 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:24.884945 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe991af8272948a4b07467542a5e1383" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a"
I20260619 01:00:24.884959 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe991af8272948a4b07467542a5e1383" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1"
I20260619 01:00:24.885124 8263 raft_consensus.cc:3037] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:24.885142 8403 raft_consensus.cc:3037] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:24.887379 8403 raft_consensus.cc:2454] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 1.
I20260619 01:00:24.887563 8263 raft_consensus.cc:2454] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 1.
I20260619 01:00:24.887781 8053 leader_election.cc:302] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:24.888123 8480 raft_consensus.cc:2781] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:24.888499 8480 raft_consensus.cc:686] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 LEADER]: Becoming Leader. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Running, Role: LEADER
I20260619 01:00:24.888933 8480 consensus_queue.cc:237] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } }
I20260619 01:00:24.893548 7962 catalog_manager.cc:5495] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0d5b7badfda34f5382c6a02d481b7537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } health_report { overall_health: HEALTHY } } }
I20260619 01:00:24.935278 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:24.929842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.0.97.1:38425)
0619 01:00:24.930046 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:24.930053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:24.930072 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:24.930329 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:24.930333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:24.930416 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:24.930566 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:24.930962 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.931739 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.933108 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:24.933463 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:24.933565 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:24.934046 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:24.934681 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:24.934870 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:24.934975 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 01:00:24.935063 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:00:24.943406 8263 raft_consensus.cc:1261] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Refusing update from remote peer 0d5b7badfda34f5382c6a02d481b7537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:24.943516 8403 raft_consensus.cc:1261] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Refusing update from remote peer 0d5b7badfda34f5382c6a02d481b7537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:24.944474 8480 consensus_queue.cc:1035] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:24.944787 8483 consensus_queue.cc:1035] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:24.955418 8486 mvcc.cc:205] Tried to move back new op lower bound from 7298379058958655488 to 7298379058748473344. Current Snapshot: MvccSnapshot[applied={T|T < 7298379058958655488}]
I20260619 01:00:24.967494 7961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:24.967746 7961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:24.971170 7961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Sending DeleteTablet for 3 replicas of tablet fe991af8272948a4b07467542a5e1383
I20260619 01:00:24.972218 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet fe991af8272948a4b07467542a5e1383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 01:00:24.972295 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet fe991af8272948a4b07467542a5e1383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47166
I20260619 01:00:24.972463 8100 tablet_service.cc:1514] Processing DeleteTablet for tablet fe991af8272948a4b07467542a5e1383 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:24 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52370
I20260619 01:00:24.974637 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:24.975658 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:24.985092 8238 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0eebf01e194ad089f29daee1f84235 (DEFAULT_TABLE table=test-table [id=56491138a4444296afa38f41aa0f278f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.985873 8238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0eebf01e194ad089f29daee1f84235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.988020 8496 tablet_replica.cc:330] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: stopping tablet replica
I20260619 01:00:24.988626 8496 raft_consensus.cc:2229] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:24.989169 8496 raft_consensus.cc:2258] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:24.991201 8496 ts_tablet_manager.cc:1905] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:24.991896 8099 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0eebf01e194ad089f29daee1f84235 (DEFAULT_TABLE table=test-table [id=56491138a4444296afa38f41aa0f278f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.992570 8099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0eebf01e194ad089f29daee1f84235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.996129 8378 tablet_service.cc:1467] Processing CreateTablet for tablet 5a0eebf01e194ad089f29daee1f84235 (DEFAULT_TABLE table=test-table [id=56491138a4444296afa38f41aa0f278f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:24.996676 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a0eebf01e194ad089f29daee1f84235. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:24.998504 8496 ts_tablet_manager.cc:1918] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:24.998769 8496 log.cc:1196] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/fe991af8272948a4b07467542a5e1383
I20260619 01:00:24.999372 8496 ts_tablet_manager.cc:1939] T fe991af8272948a4b07467542a5e1383 P 3998af71bd8249f2a5565c082539bea1: Deleting consensus metadata
I20260619 01:00:25.003544 7950 catalog_manager.cc:4841] TS 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293): tablet fe991af8272948a4b07467542a5e1383 (table test-table [id=83fc9f28d367458f86698935316fec38]) successfully deleted
I20260619 01:00:25.010257 8472 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Bootstrap starting.
I20260619 01:00:25.011902 8472 tablet_bootstrap.cc:654] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.015938 8473 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Bootstrap starting.
I20260619 01:00:25.017546 8473 tablet_bootstrap.cc:654] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.023005 8474 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Bootstrap starting.
I20260619 01:00:25.025064 8474 tablet_bootstrap.cc:654] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.031777 8472 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: No bootstrap required, opened a new log
I20260619 01:00:25.031919 8472 ts_tablet_manager.cc:1397] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.008s
I20260619 01:00:25.032652 8472 raft_consensus.cc:348] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.032879 8472 raft_consensus.cc:374] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.032967 8472 raft_consensus.cc:729] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.034782 8472 consensus_queue.cc:260] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.037261 8472 ts_tablet_manager.cc:1428] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260619 01:00:25.039947 8474 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: No bootstrap required, opened a new log
I20260619 01:00:25.040155 8474 ts_tablet_manager.cc:1397] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20260619 01:00:25.041328 8474 raft_consensus.cc:348] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.041630 8474 raft_consensus.cc:374] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.041750 8474 raft_consensus.cc:729] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.042068 8474 consensus_queue.cc:260] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.043007 8474 ts_tablet_manager.cc:1428] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:25.043706 8497 tablet_replica.cc:330] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: stopping tablet replica
I20260619 01:00:25.044718 8497 raft_consensus.cc:2229] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:25.045667 8497 raft_consensus.cc:2258] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.048267 8497 ts_tablet_manager.cc:1905] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.056221 8473 tablet_bootstrap.cc:492] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: No bootstrap required, opened a new log
I20260619 01:00:25.056375 8495 tablet_replica.cc:330] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: stopping tablet replica
I20260619 01:00:25.057143 8497 ts_tablet_manager.cc:1918] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:25.056439 8473 ts_tablet_manager.cc:1397] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.003s
I20260619 01:00:25.057332 8497 log.cc:1196] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/fe991af8272948a4b07467542a5e1383
I20260619 01:00:25.057806 8497 ts_tablet_manager.cc:1939] T fe991af8272948a4b07467542a5e1383 P 0d5b7badfda34f5382c6a02d481b7537: Deleting consensus metadata
I20260619 01:00:25.057796 8495 raft_consensus.cc:2229] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:25.058079 8473 raft_consensus.cc:348] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.058290 8473 raft_consensus.cc:374] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.058396 8473 raft_consensus.cc:729] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.058537 8495 raft_consensus.cc:2258] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.058636 8473 consensus_queue.cc:260] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.059180 8473 ts_tablet_manager.cc:1428] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:25.059381 7947 catalog_manager.cc:4841] TS 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425): tablet fe991af8272948a4b07467542a5e1383 (table test-table [id=83fc9f28d367458f86698935316fec38]) successfully deleted
I20260619 01:00:25.060760 8495 ts_tablet_manager.cc:1905] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.065449 8495 ts_tablet_manager.cc:1918] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:25.065626 8495 log.cc:1196] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/fe991af8272948a4b07467542a5e1383
I20260619 01:00:25.066030 8495 ts_tablet_manager.cc:1939] T fe991af8272948a4b07467542a5e1383 P 1267de06ed2847b88b5a3bad45602b6a: Deleting consensus metadata
I20260619 01:00:25.067440 7949 catalog_manager.cc:4841] TS 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043): tablet fe991af8272948a4b07467542a5e1383 (table test-table [id=83fc9f28d367458f86698935316fec38]) successfully deleted
I20260619 01:00:25.132580 8483 raft_consensus.cc:482] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:25.132869 8483 raft_consensus.cc:504] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.133658 8483 leader_election.cc:288] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.134203 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0eebf01e194ad089f29daee1f84235" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a" is_pre_election: true
I20260619 01:00:25.134238 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0eebf01e194ad089f29daee1f84235" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1" is_pre_election: true
I20260619 01:00:25.134465 8403 raft_consensus.cc:2454] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 0.
I20260619 01:00:25.134460 8263 raft_consensus.cc:2454] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 0.
I20260619 01:00:25.134853 8053 leader_election.cc:302] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.135124 8483 raft_consensus.cc:2781] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:25.135238 8483 raft_consensus.cc:482] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:25.135346 8483 raft_consensus.cc:3037] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.138552 8483 raft_consensus.cc:504] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.139259 8483 leader_election.cc:288] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 election: Requested vote from peers 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.139650 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0eebf01e194ad089f29daee1f84235" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1"
I20260619 01:00:25.139851 8403 raft_consensus.cc:3037] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.140888 8263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a0eebf01e194ad089f29daee1f84235" candidate_uuid: "0d5b7badfda34f5382c6a02d481b7537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a"
I20260619 01:00:25.141112 8263 raft_consensus.cc:3037] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.142954 8403 raft_consensus.cc:2454] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 1.
I20260619 01:00:25.143141 8263 raft_consensus.cc:2454] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5b7badfda34f5382c6a02d481b7537 in term 1.
I20260619 01:00:25.143359 8053 leader_election.cc:302] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.143653 8483 raft_consensus.cc:2781] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:25.143846 8483 raft_consensus.cc:686] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 LEADER]: Becoming Leader. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Running, Role: LEADER
I20260619 01:00:25.144098 8483 consensus_queue.cc:237] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.147022 7961 catalog_manager.cc:5495] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "0d5b7badfda34f5382c6a02d481b7537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } health_report { overall_health: UNKNOWN } } }
Jun 19 01:00:25 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830822, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:25.202832 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.189147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.0.97.62:44169)
0619 01:00:25.189804 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.189810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.190021 (+ 211us) server_negotiation.cc:407] Connection header received
0619 01:00:25.190221 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.190228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.190295 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.190414 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:25.191552 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.192281 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.193490 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.193904 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.194041 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.197560 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:25.197564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:25.197568 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:25.197603 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:25.200186 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:25.200852 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:25.200859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:25.200865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:25.200981 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:25.201357 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:25.201364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:25.201369 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:25.201623 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:25.201766 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.202150 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.202384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":469,"thread_start_us":296,"threads_started":1}
I20260619 01:00:25.202886 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.189137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44169 (local address 127.0.0.1:55098)
0619 01:00:25.189911 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.189926 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.190103 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.190447 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.190453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.191218 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:25.191400 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.191410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.192414 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.192422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.193352 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.193363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.194073 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.194078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.194241 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.194978 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:25.195001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:25.197360 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:25.200318 (+ 2958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:25.200330 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:25.200338 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:25.200733 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:25.201101 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:25.201106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:25.201109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:25.201256 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:25.201754 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:25.201762 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:25.201908 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.202090 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.202509 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":152,"thread_start_us":497,"threads_started":1}
I20260619 01:00:25.208338 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55098:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 01:00:25.209311 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:25.216226 8100 tablet_service.cc:1467] Processing CreateTablet for tablet 0e30ed783f8b48d988e7e34ec7ff5184 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d822a9a77bd44ea9a227e88f838227b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:00:25.216566 8239 tablet_service.cc:1467] Processing CreateTablet for tablet 0e30ed783f8b48d988e7e34ec7ff5184 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d822a9a77bd44ea9a227e88f838227b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:00:25.216897 8100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e30ed783f8b48d988e7e34ec7ff5184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.217058 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e30ed783f8b48d988e7e34ec7ff5184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.218977 8378 tablet_service.cc:1467] Processing CreateTablet for tablet 0e30ed783f8b48d988e7e34ec7ff5184 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6d822a9a77bd44ea9a227e88f838227b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 01:00:25.219538 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e30ed783f8b48d988e7e34ec7ff5184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.224174 8474 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: Bootstrap starting.
I20260619 01:00:25.224187 8473 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: Bootstrap starting.
I20260619 01:00:25.225773 8474 tablet_bootstrap.cc:654] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.225884 8473 tablet_bootstrap.cc:654] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.227897 8474 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: No bootstrap required, opened a new log
I20260619 01:00:25.228089 8474 ts_tablet_manager.cc:1397] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 01:00:25.228683 8474 raft_consensus.cc:348] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.228909 8474 raft_consensus.cc:374] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.229022 8474 raft_consensus.cc:729] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.229293 8474 consensus_queue.cc:260] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.229899 8483 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.230168 8483 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.230926 8474 ts_tablet_manager.cc:1428] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 01:00:25.232197 8472 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: Bootstrap starting.
I20260619 01:00:25.233799 8472 tablet_bootstrap.cc:654] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.235873 8472 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: No bootstrap required, opened a new log
I20260619 01:00:25.236048 8472 ts_tablet_manager.cc:1397] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260619 01:00:25.236753 8472 raft_consensus.cc:348] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.236991 8472 raft_consensus.cc:374] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.237073 8472 raft_consensus.cc:729] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.237301 8472 consensus_queue.cc:260] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.237839 8472 ts_tablet_manager.cc:1428] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 01:00:25.237787 8478 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.238116 8478 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.238998 8473 tablet_bootstrap.cc:492] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: No bootstrap required, opened a new log
I20260619 01:00:25.239221 8473 ts_tablet_manager.cc:1397] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20260619 01:00:25.240113 8473 raft_consensus.cc:348] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.240342 8473 raft_consensus.cc:374] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.240432 8473 raft_consensus.cc:729] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.240752 8473 consensus_queue.cc:260] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.241411 8473 ts_tablet_manager.cc:1428] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:25.241271 8479 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.241668 8479 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.307040 8479 raft_consensus.cc:482] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:25.307273 8479 raft_consensus.cc:504] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.309046 8479 leader_election.cc:288] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.317087 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.309401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48157 (local address 127.0.97.1:38425)
0619 01:00:25.309590 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.309596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.310016 (+ 420us) server_negotiation.cc:407] Connection header received
0619 01:00:25.310284 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.310289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.310357 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.310461 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:25.311442 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.312594 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.314981 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.315787 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.315874 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.315986 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.316630 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.316764 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:00:25.317203 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.309230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38425 (local address 127.0.97.2:48157)
0619 01:00:25.309910 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.309929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.310123 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.310865 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.310868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.310899 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:25.311273 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.311282 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.312735 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.312740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.314718 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.314790 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.316180 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.316185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.316412 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.316470 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.316561 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.316731 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":91,"thread_start_us":617,"threads_started":1}
I20260619 01:00:25.317239 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.309256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45293 (local address 127.0.97.2:41231)
0619 01:00:25.310111 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.310136 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.310340 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.310743 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.310748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.310780 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:25.311171 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.311187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.312273 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.312279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.314399 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.314413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.316098 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.316106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.316358 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.316468 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.316561 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.316731 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":68,"thread_start_us":752,"threads_started":1}
I20260619 01:00:25.318022 8470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.309651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41231 (local address 127.0.97.3:45293)
0619 01:00:25.309841 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.309848 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.310231 (+ 383us) server_negotiation.cc:407] Connection header received
0619 01:00:25.310500 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.310507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.310588 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.310671 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:25.311383 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.312133 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.314578 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.315865 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.315998 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.316091 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.317631 (+ 1540us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.317764 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:00:25.318619 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e30ed783f8b48d988e7e34ec7ff5184" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1" is_pre_election: true
I20260619 01:00:25.318925 8403 raft_consensus.cc:2454] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 0.
I20260619 01:00:25.319031 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e30ed783f8b48d988e7e34ec7ff5184" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537" is_pre_election: true
I20260619 01:00:25.319447 8120 raft_consensus.cc:2454] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 0.
I20260619 01:00:25.319520 8193 leader_election.cc:302] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1267de06ed2847b88b5a3bad45602b6a, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.319952 8479 raft_consensus.cc:2781] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:25.320127 8479 raft_consensus.cc:482] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:25.320225 8479 raft_consensus.cc:3037] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.322280 8479 raft_consensus.cc:504] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.322852 8479 leader_election.cc:288] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 election: Requested vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.323226 8120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e30ed783f8b48d988e7e34ec7ff5184" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537"
I20260619 01:00:25.323319 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e30ed783f8b48d988e7e34ec7ff5184" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1"
I20260619 01:00:25.323426 8120 raft_consensus.cc:3037] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.323527 8403 raft_consensus.cc:3037] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.326327 8403 raft_consensus.cc:2454] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 1.
I20260619 01:00:25.326678 8193 leader_election.cc:302] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1267de06ed2847b88b5a3bad45602b6a, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.326972 8479 raft_consensus.cc:2781] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:25.327191 8120 raft_consensus.cc:2454] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 1.
I20260619 01:00:25.327693 8479 raft_consensus.cc:686] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [term 1 LEADER]: Becoming Leader. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Running, Role: LEADER
I20260619 01:00:25.328084 8479 consensus_queue.cc:237] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.330070 8514 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: TxnStatusTablet state changed. Reason: New leader 1267de06ed2847b88b5a3bad45602b6a. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.330461 8514 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:00:25.331225 8516 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 01:00:25.331475 8516 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 01:00:25.331874 7962 catalog_manager.cc:5495] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } health_report { overall_health: UNKNOWN } } }
I20260619 01:00:25.365108 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.358737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43274 (local address 127.0.97.2:38043)
0619 01:00:25.358899 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.358905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.359135 (+ 230us) server_negotiation.cc:407] Connection header received
0619 01:00:25.359429 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.359433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.359528 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.359683 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.360301 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.361317 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.362765 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.363326 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.363456 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.363952 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.364452 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.364605 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.364691 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.364764 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 01:00:25.365108 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.358721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.0.1:43274)
0619 01:00:25.359033 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.359049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.359226 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.359895 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.359898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.359907 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:00:25.360115 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.360138 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.361441 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.361448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.362625 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.362643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.363467 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.363472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.363676 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.363761 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.364567 (+ 806us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.364593 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.364674 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.364812 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260619 01:00:25.369169 8403 raft_consensus.cc:1261] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Refusing update from remote peer 1267de06ed2847b88b5a3bad45602b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.369503 8120 raft_consensus.cc:1261] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Refusing update from remote peer 1267de06ed2847b88b5a3bad45602b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.369906 8479 consensus_queue.cc:1035] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.370410 8514 consensus_queue.cc:1035] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.383185 8483 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: TxnStatusTablet state changed. Reason: New leader 1267de06ed2847b88b5a3bad45602b6a. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.383436 8483 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.384492 8478 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: TxnStatusTablet state changed. Reason: New leader 1267de06ed2847b88b5a3bad45602b6a. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.384743 8478 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.386350 8514 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.386598 8514 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:00:25.386839 8488 mvcc.cc:205] Tried to move back new op lower bound from 7298379060705120256 to 7298379060547432448. Current Snapshot: MvccSnapshot[applied={T|T < 7298379060705120256 or (T in {7298379060705120256})}]
I20260619 01:00:25.390208 8483 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.390434 8483 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 0d5b7badfda34f5382c6a02d481b7537: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.394392 8479 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.394637 8479 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 1267de06ed2847b88b5a3bad45602b6a: This TxnStatusTablet replica's current role is: LEADER
I20260619 01:00:25.395723 8487 mvcc.cc:205] Tried to move back new op lower bound from 7298379060705120256 to 7298379060547432448. Current Snapshot: MvccSnapshot[applied={T|T < 7298379060705120256}]
I20260619 01:00:25.397109 8478 tablet_replica.cc:439] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } } }
I20260619 01:00:25.397324 8478 tablet_replica.cc:442] T 0e30ed783f8b48d988e7e34ec7ff5184 P 3998af71bd8249f2a5565c082539bea1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 01:00:25.411852 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.406391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53857 (local address 127.0.97.2:38043)
0619 01:00:25.406573 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.406579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.406889 (+ 310us) server_negotiation.cc:407] Connection header received
0619 01:00:25.407111 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.407115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.407188 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.407299 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.407765 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.408601 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.410029 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.410469 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.410606 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.411147 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.411242 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.411340 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.411483 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.411553 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:00:25.412177 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.406058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.97.1:53857)
0619 01:00:25.406778 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.406796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.406961 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.407313 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.407317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.407328 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:00:25.407611 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.407619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.408723 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.408730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.409876 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.409887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.410665 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.410672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.410866 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.410923 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.411341 (+ 418us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.411357 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.411454 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.411593 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":527,"threads_started":1}
I20260619 01:00:25.421491 8263 raft_consensus.cc:1261] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Refusing update from remote peer 0d5b7badfda34f5382c6a02d481b7537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.422068 8480 consensus_queue.cc:1035] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.422504 8403 raft_consensus.cc:1261] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Refusing update from remote peer 0d5b7badfda34f5382c6a02d481b7537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.423079 8480 consensus_queue.cc:1035] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.482285 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.466745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44675 (local address 127.0.97.1:38425)
0619 01:00:25.466897 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.466903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.476335 (+ 9432us) server_negotiation.cc:407] Connection header received
0619 01:00:25.476564 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.476568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.476625 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.476714 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.477225 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.477953 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.480471 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.480949 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.481113 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.481602 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.481691 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.481785 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.481905 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.481976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 01:00:25.482427 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.466432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38425 (local address 127.0.97.2:44675)
0619 01:00:25.476227 (+ 9795us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.476244 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.476409 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.476726 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.476731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.476741 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:00:25.477086 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.477094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.478073 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.478080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.480308 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.480321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.481149 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.481156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.481357 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.481436 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.481785 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.481809 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.481879 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.482018 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9630,"thread_start_us":9479,"threads_started":1}
I20260619 01:00:25.543537 7961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:25.543751 7961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:25.546713 7961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Sending DeleteTablet for 3 replicas of tablet 5a0eebf01e194ad089f29daee1f84235
I20260619 01:00:25.547451 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0eebf01e194ad089f29daee1f84235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 01:00:25.547607 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0eebf01e194ad089f29daee1f84235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47166
I20260619 01:00:25.547717 8495 tablet_replica.cc:330] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: stopping tablet replica
I20260619 01:00:25.547910 8495 raft_consensus.cc:2229] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:25.548115 8495 raft_consensus.cc:2258] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.548272 8531 tablet_replica.cc:330] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: stopping tablet replica
I20260619 01:00:25.548507 8100 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a0eebf01e194ad089f29daee1f84235 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52370
I20260619 01:00:25.548507 8531 raft_consensus.cc:2229] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:25.548740 8497 tablet_replica.cc:330] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: stopping tablet replica
I20260619 01:00:25.548902 8531 raft_consensus.cc:2258] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.548929 8497 raft_consensus.cc:2229] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:25.549211 8497 raft_consensus.cc:2258] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.549324 8495 ts_tablet_manager.cc:1905] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.550005 8531 ts_tablet_manager.cc:1905] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.550163 8497 ts_tablet_manager.cc:1905] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.556336 8495 ts_tablet_manager.cc:1918] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:25.556496 8495 log.cc:1196] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/5a0eebf01e194ad089f29daee1f84235
I20260619 01:00:25.556803 8531 ts_tablet_manager.cc:1918] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:25.556959 8495 ts_tablet_manager.cc:1939] T 5a0eebf01e194ad089f29daee1f84235 P 1267de06ed2847b88b5a3bad45602b6a: Deleting consensus metadata
I20260619 01:00:25.557067 8531 log.cc:1196] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/5a0eebf01e194ad089f29daee1f84235
I20260619 01:00:25.557530 8531 ts_tablet_manager.cc:1939] T 5a0eebf01e194ad089f29daee1f84235 P 3998af71bd8249f2a5565c082539bea1: Deleting consensus metadata
I20260619 01:00:25.558037 7949 catalog_manager.cc:4841] TS 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043): tablet 5a0eebf01e194ad089f29daee1f84235 (table test-table [id=56491138a4444296afa38f41aa0f278f]) successfully deleted
I20260619 01:00:25.558501 7950 catalog_manager.cc:4841] TS 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293): tablet 5a0eebf01e194ad089f29daee1f84235 (table test-table [id=56491138a4444296afa38f41aa0f278f]) successfully deleted
I20260619 01:00:25.565716 8497 ts_tablet_manager.cc:1918] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:25.565922 8497 log.cc:1196] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/5a0eebf01e194ad089f29daee1f84235
I20260619 01:00:25.566356 8497 ts_tablet_manager.cc:1939] T 5a0eebf01e194ad089f29daee1f84235 P 0d5b7badfda34f5382c6a02d481b7537: Deleting consensus metadata
I20260619 01:00:25.567373 7947 catalog_manager.cc:4841] TS 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425): tablet 5a0eebf01e194ad089f29daee1f84235 (table test-table [id=56491138a4444296afa38f41aa0f278f]) successfully deleted
Jun 19 01:00:25 dist-test-slave-3rxd krb5kdc[7918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 01:00:25 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:25 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:00:25.611356 8507 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:25.600481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.97.62:44169)
0619 01:00:25.600722 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.600731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.601112 (+ 381us) server_negotiation.cc:407] Connection header received
0619 01:00:25.601319 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.601323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.601385 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.601462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:25.603107 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.603807 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.605588 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.605967 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.606053 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.611101 (+ 5048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55112: BlockingRecv error: recv got EOF from 127.0.0.1:55112 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 01:00:25 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260619 01:00:25.628624 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.613949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.0.97.62:44169)
0619 01:00:25.614153 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.614161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.614619 (+ 458us) server_negotiation.cc:407] Connection header received
0619 01:00:25.614760 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.614766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.614837 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.614936 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:25.616059 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.617071 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.618425 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.619137 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.619294 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.622782 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:25.622790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:25.622795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:25.622830 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:25.626052 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:25.626700 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:25.626706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:25.626709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:25.626884 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:25.627222 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:25.627227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:25.627230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:25.627549 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:25.627802 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.628025 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.628265 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:25.631876 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:25.632498 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:25.638653 8100 tablet_service.cc:1467] Processing CreateTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (DEFAULT_TABLE table=test-table [id=b1ab92fefef748c8a01ea2f09d040488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.639092 8100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85f11db8f66420ba5ef2f3c7f4c60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.639071 8239 tablet_service.cc:1467] Processing CreateTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (DEFAULT_TABLE table=test-table [id=b1ab92fefef748c8a01ea2f09d040488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.639062 8378 tablet_service.cc:1467] Processing CreateTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (DEFAULT_TABLE table=test-table [id=b1ab92fefef748c8a01ea2f09d040488]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.639432 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85f11db8f66420ba5ef2f3c7f4c60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.639427 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e85f11db8f66420ba5ef2f3c7f4c60d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.646010 8472 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Bootstrap starting.
I20260619 01:00:25.646039 8473 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Bootstrap starting.
I20260619 01:00:25.646428 8474 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Bootstrap starting.
I20260619 01:00:25.647622 8473 tablet_bootstrap.cc:654] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.647619 8472 tablet_bootstrap.cc:654] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.647909 8474 tablet_bootstrap.cc:654] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.656876 8474 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: No bootstrap required, opened a new log
I20260619 01:00:25.657023 8474 ts_tablet_manager.cc:1397] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260619 01:00:25.657825 8474 raft_consensus.cc:348] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.658089 8474 raft_consensus.cc:374] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.658186 8474 raft_consensus.cc:729] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.658409 8474 consensus_queue.cc:260] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.659030 8472 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: No bootstrap required, opened a new log
I20260619 01:00:25.659241 8472 ts_tablet_manager.cc:1397] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260619 01:00:25.660212 8472 raft_consensus.cc:348] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.660583 8472 raft_consensus.cc:374] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.660737 8472 raft_consensus.cc:729] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.661098 8472 consensus_queue.cc:260] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.662268 8474 ts_tablet_manager.cc:1428] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260619 01:00:25.663013 8472 ts_tablet_manager.cc:1428] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 01:00:25.664629 8473 tablet_bootstrap.cc:492] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: No bootstrap required, opened a new log
I20260619 01:00:25.664780 8473 ts_tablet_manager.cc:1397] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.006s
I20260619 01:00:25.665513 8473 raft_consensus.cc:348] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.665784 8473 raft_consensus.cc:374] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.665915 8473 raft_consensus.cc:729] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.666150 8473 consensus_queue.cc:260] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.666657 8473 ts_tablet_manager.cc:1428] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 01:00:25.724773 8514 raft_consensus.cc:482] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:25.725028 8514 raft_consensus.cc:504] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.725850 8514 leader_election.cc:288] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.726366 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85f11db8f66420ba5ef2f3c7f4c60d4" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537" is_pre_election: true
I20260619 01:00:25.726452 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85f11db8f66420ba5ef2f3c7f4c60d4" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1" is_pre_election: true
I20260619 01:00:25.726600 8119 raft_consensus.cc:2454] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 0.
I20260619 01:00:25.726802 8403 raft_consensus.cc:2454] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 0.
I20260619 01:00:25.727293 8193 leader_election.cc:302] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 1267de06ed2847b88b5a3bad45602b6a, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.727685 8514 raft_consensus.cc:2781] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:25.727788 8514 raft_consensus.cc:482] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:25.727910 8514 raft_consensus.cc:3037] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.729993 8514 raft_consensus.cc:504] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.730532 8514 leader_election.cc:288] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 election: Requested vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293)
I20260619 01:00:25.731029 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85f11db8f66420ba5ef2f3c7f4c60d4" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3998af71bd8249f2a5565c082539bea1"
I20260619 01:00:25.731071 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e85f11db8f66420ba5ef2f3c7f4c60d4" candidate_uuid: "1267de06ed2847b88b5a3bad45602b6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537"
I20260619 01:00:25.731237 8403 raft_consensus.cc:3037] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.731307 8119 raft_consensus.cc:3037] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.733251 8403 raft_consensus.cc:2454] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 1.
I20260619 01:00:25.733630 8193 leader_election.cc:302] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1267de06ed2847b88b5a3bad45602b6a, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.733880 8514 raft_consensus.cc:2781] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:25.734061 8514 raft_consensus.cc:686] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 1 LEADER]: Becoming Leader. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Running, Role: LEADER
I20260619 01:00:25.734319 8514 consensus_queue.cc:237] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.734642 8119 raft_consensus.cc:2454] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1267de06ed2847b88b5a3bad45602b6a in term 1.
I20260619 01:00:25.740432 7961 catalog_manager.cc:5495] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1267de06ed2847b88b5a3bad45602b6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } health_report { overall_health: UNKNOWN } } }
I20260619 01:00:25.757447 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.751165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.0.97.2:38043)
0619 01:00:25.751829 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.751836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.751855 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:25.751920 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.751925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.752010 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.752121 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.752461 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.753844 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.755316 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.756099 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.756258 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.756600 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.756779 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.756886 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.757031 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.757113 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260619 01:00:25.760986 8403 raft_consensus.cc:1261] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Refusing update from remote peer 1267de06ed2847b88b5a3bad45602b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.761029 8119 raft_consensus.cc:1261] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Refusing update from remote peer 1267de06ed2847b88b5a3bad45602b6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.761569 8514 consensus_queue.cc:1035] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.761879 8529 consensus_queue.cc:1035] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.784936 7961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:25.785161 7961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:25.788264 7961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Sending DeleteTablet for 3 replicas of tablet e85f11db8f66420ba5ef2f3c7f4c60d4
I20260619 01:00:25.789041 8100 tablet_service.cc:1514] Processing DeleteTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52370
I20260619 01:00:25.789048 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 01:00:25.789320 8497 tablet_replica.cc:330] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: stopping tablet replica
I20260619 01:00:25.789346 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet e85f11db8f66420ba5ef2f3c7f4c60d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47166
I20260619 01:00:25.789489 8497 raft_consensus.cc:2229] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:25.789614 8495 tablet_replica.cc:330] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: stopping tablet replica
I20260619 01:00:25.789615 8531 tablet_replica.cc:330] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: stopping tablet replica
I20260619 01:00:25.789747 8497 raft_consensus.cc:2258] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.789795 8495 raft_consensus.cc:2229] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:25.789840 8531 raft_consensus.cc:2229] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:25.790059 8531 raft_consensus.cc:2258] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.790311 8495 raft_consensus.cc:2258] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:25.790953 8497 ts_tablet_manager.cc:1905] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.791724 8495 ts_tablet_manager.cc:1905] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.791918 7961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:25.792670 7961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:25.798136 8495 ts_tablet_manager.cc:1918] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:25.798269 8495 log.cc:1196] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/e85f11db8f66420ba5ef2f3c7f4c60d4
I20260619 01:00:25.798983 8495 ts_tablet_manager.cc:1939] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 1267de06ed2847b88b5a3bad45602b6a: Deleting consensus metadata
I20260619 01:00:25.799986 8100 tablet_service.cc:1467] Processing CreateTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 (DEFAULT_TABLE table=test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.800050 8378 tablet_service.cc:1467] Processing CreateTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 (DEFAULT_TABLE table=test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.800410 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc132ff26434ddeb0e4b15c3abc57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.800417 8100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc132ff26434ddeb0e4b15c3abc57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.802091 8497 ts_tablet_manager.cc:1918] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:25.802246 8497 log.cc:1196] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/e85f11db8f66420ba5ef2f3c7f4c60d4
I20260619 01:00:25.802757 8497 ts_tablet_manager.cc:1939] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 0d5b7badfda34f5382c6a02d481b7537: Deleting consensus metadata
I20260619 01:00:25.804219 7947 catalog_manager.cc:4841] TS 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425): tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (table test-table [id=b1ab92fefef748c8a01ea2f09d040488]) successfully deleted
I20260619 01:00:25.805189 8239 tablet_service.cc:1467] Processing CreateTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 (DEFAULT_TABLE table=test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:25.805667 8239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edc132ff26434ddeb0e4b15c3abc57d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:25.807168 8472 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Bootstrap starting.
I20260619 01:00:25.808666 8472 tablet_bootstrap.cc:654] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.810392 8474 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Bootstrap starting.
I20260619 01:00:25.812080 8474 tablet_bootstrap.cc:654] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.813980 8531 ts_tablet_manager.cc:1905] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:25.816912 8473 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Bootstrap starting.
I20260619 01:00:25.818473 8474 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: No bootstrap required, opened a new log
I20260619 01:00:25.819037 8474 ts_tablet_manager.cc:1397] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 01:00:25.819195 8531 ts_tablet_manager.cc:1918] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:25.819319 8531 log.cc:1196] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/e85f11db8f66420ba5ef2f3c7f4c60d4
I20260619 01:00:25.819804 8531 ts_tablet_manager.cc:1939] T e85f11db8f66420ba5ef2f3c7f4c60d4 P 3998af71bd8249f2a5565c082539bea1: Deleting consensus metadata
I20260619 01:00:25.819867 8473 tablet_bootstrap.cc:654] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:25.819850 8474 raft_consensus.cc:348] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.820130 8474 raft_consensus.cc:374] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.820227 8474 raft_consensus.cc:729] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5b7badfda34f5382c6a02d481b7537, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.820502 8474 consensus_queue.cc:260] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.820881 7950 catalog_manager.cc:4841] TS 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293): tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (table test-table [id=b1ab92fefef748c8a01ea2f09d040488]) successfully deleted
I20260619 01:00:25.821350 7949 catalog_manager.cc:4841] TS 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043): tablet e85f11db8f66420ba5ef2f3c7f4c60d4 (table test-table [id=b1ab92fefef748c8a01ea2f09d040488]) successfully deleted
I20260619 01:00:25.822463 8473 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: No bootstrap required, opened a new log
I20260619 01:00:25.822759 8473 ts_tablet_manager.cc:1397] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 01:00:25.823639 8474 ts_tablet_manager.cc:1428] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 01:00:25.824760 8473 raft_consensus.cc:348] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.825094 8473 raft_consensus.cc:374] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.825234 8473 raft_consensus.cc:729] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1267de06ed2847b88b5a3bad45602b6a, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.825567 8473 consensus_queue.cc:260] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.825824 8472 tablet_bootstrap.cc:492] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: No bootstrap required, opened a new log
I20260619 01:00:25.826607 8473 ts_tablet_manager.cc:1428] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 01:00:25.826601 8472 ts_tablet_manager.cc:1397] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20260619 01:00:25.827382 8472 raft_consensus.cc:348] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.827646 8472 raft_consensus.cc:374] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:25.827729 8472 raft_consensus.cc:729] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Initialized, Role: FOLLOWER
I20260619 01:00:25.827958 8472 consensus_queue.cc:260] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.828413 8472 ts_tablet_manager.cc:1428] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 01:00:25.842423 8478 raft_consensus.cc:482] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:25.842620 8478 raft_consensus.cc:504] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.844453 8478 leader_election.cc:288] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043)
I20260619 01:00:25.853430 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.845201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53507 (local address 127.0.97.1:38425)
0619 01:00:25.845419 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.845425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.845993 (+ 568us) server_negotiation.cc:407] Connection header received
0619 01:00:25.846801 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.846809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.846881 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.846958 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:25.847837 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.848899 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.851733 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.852514 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.852658 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.852751 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.853024 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.853156 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:00:25.853751 8554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.844585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38425 (local address 127.0.97.3:53507)
0619 01:00:25.845889 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.845910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.846628 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.847087 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.847090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.847131 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:25.847592 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.847627 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.849063 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.849068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.851552 (+ 2484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.851561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.852686 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.852692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.852871 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.852894 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.852999 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.853128 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":212,"spinlock_wait_cycles":5888,"thread_start_us":1057,"threads_started":1}
I20260619 01:00:25.854415 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.844875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56347 (local address 127.0.97.2:38043)
0619 01:00:25.845068 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.845075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.846441 (+ 1366us) server_negotiation.cc:407] Connection header received
0619 01:00:25.846640 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.846646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.846765 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.846843 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:25.847652 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.848894 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.851715 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.853040 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.853166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.853289 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.853955 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.854148 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:00:25.854470 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc132ff26434ddeb0e4b15c3abc57d3" candidate_uuid: "3998af71bd8249f2a5565c082539bea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537" is_pre_election: true
I20260619 01:00:25.854729 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.844704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.97.3:56347)
0619 01:00:25.846335 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.846357 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.846512 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.847008 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.847013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.847036 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:25.847435 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.847454 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.849063 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.849068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.851537 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.851551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.853457 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.853464 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.853682 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.853726 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.853841 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.853995 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1474,"mutex_wait_us":133,"spinlock_wait_cycles":11520,"thread_start_us":1279,"threads_started":1}
I20260619 01:00:25.854733 8119 raft_consensus.cc:2454] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3998af71bd8249f2a5565c082539bea1 in term 0.
I20260619 01:00:25.855217 8329 leader_election.cc:302] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.855475 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc132ff26434ddeb0e4b15c3abc57d3" candidate_uuid: "3998af71bd8249f2a5565c082539bea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a" is_pre_election: true
I20260619 01:00:25.855626 8478 raft_consensus.cc:2781] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:25.855752 8478 raft_consensus.cc:482] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:25.855752 8258 raft_consensus.cc:2454] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3998af71bd8249f2a5565c082539bea1 in term 0.
I20260619 01:00:25.855854 8478 raft_consensus.cc:3037] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.859252 8478 raft_consensus.cc:504] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.859896 8478 leader_election.cc:288] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [CANDIDATE]: Term 1 election: Requested vote from peers 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425), 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043)
I20260619 01:00:25.860299 8119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc132ff26434ddeb0e4b15c3abc57d3" candidate_uuid: "3998af71bd8249f2a5565c082539bea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d5b7badfda34f5382c6a02d481b7537"
I20260619 01:00:25.860316 8258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edc132ff26434ddeb0e4b15c3abc57d3" candidate_uuid: "3998af71bd8249f2a5565c082539bea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1267de06ed2847b88b5a3bad45602b6a"
I20260619 01:00:25.860498 8119 raft_consensus.cc:3037] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.860576 8258 raft_consensus.cc:3037] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:25.862521 8119 raft_consensus.cc:2454] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3998af71bd8249f2a5565c082539bea1 in term 1.
I20260619 01:00:25.862989 8329 leader_election.cc:302] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5b7badfda34f5382c6a02d481b7537, 3998af71bd8249f2a5565c082539bea1; no voters:
I20260619 01:00:25.863252 8478 raft_consensus.cc:2781] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:25.863772 8478 raft_consensus.cc:686] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 1 LEADER]: Becoming Leader. State: Replica: 3998af71bd8249f2a5565c082539bea1, State: Running, Role: LEADER
I20260619 01:00:25.863865 8258 raft_consensus.cc:2454] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3998af71bd8249f2a5565c082539bea1 in term 1.
I20260619 01:00:25.864413 8478 consensus_queue.cc:237] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } }
I20260619 01:00:25.867985 7961 catalog_manager.cc:5495] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3998af71bd8249f2a5565c082539bea1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3998af71bd8249f2a5565c082539bea1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3998af71bd8249f2a5565c082539bea1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45293 } health_report { overall_health: HEALTHY } } }
I20260619 01:00:25.913852 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.898753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47170 (local address 127.0.97.3:45293)
0619 01:00:25.899329 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.899336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.906289 (+ 6953us) server_negotiation.cc:407] Connection header received
0619 01:00:25.906523 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.906529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.906633 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.906859 (+ 226us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.907259 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.908542 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.910455 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.911215 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.911401 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.911922 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.912737 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.913312 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.913379 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.913458 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":553,"threads_started":1}
I20260619 01:00:25.929493 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.924196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47277 (local address 127.0.97.2:38043)
0619 01:00:25.924338 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.924343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.924924 (+ 581us) server_negotiation.cc:407] Connection header received
0619 01:00:25.925122 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.925129 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.925205 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.925289 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.925721 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.926527 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.927823 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.928239 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.928378 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.928765 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.928878 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.928975 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.929113 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.929180 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 01:00:25.929533 8561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.924209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38043 (local address 127.0.97.3:47277)
0619 01:00:25.924836 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.924851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.924984 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.925313 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.925318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.925327 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:00:25.925587 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.925595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.926656 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.926664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.927690 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.927700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.928400 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.928406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.928546 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.928589 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.928974 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.928987 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.929078 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.929194 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
I20260619 01:00:25.937366 8258 raft_consensus.cc:1261] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Refusing update from remote peer 3998af71bd8249f2a5565c082539bea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.937428 8119 raft_consensus.cc:1261] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Refusing update from remote peer 3998af71bd8249f2a5565c082539bea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:25.938056 8556 consensus_queue.cc:1035] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d5b7badfda34f5382c6a02d481b7537" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.938452 8478 consensus_queue.cc:1035] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1267de06ed2847b88b5a3bad45602b6a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:25.968375 8558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.961404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38087 (local address 127.0.97.3:45293)
0619 01:00:25.961570 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:25.961576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:25.961594 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:00:25.961660 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:25.961665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:25.961759 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:25.961843 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:25.962523 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.963717 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.966504 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.967191 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.967310 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.967803 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.967890 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.967956 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:25.968114 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 01:00:25.968167 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:00:25.968587 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:25.960638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45293 (local address 127.0.97.2:38087)
0619 01:00:25.961062 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:25.961084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:25.961202 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:25.962079 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:25.962090 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:25.962105 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 01:00:25.962324 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.962336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.964527 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.964533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.966320 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:25.966335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:25.967425 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:25.967431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:25.967618 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:25.967663 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 01:00:25.967998 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 01:00:25.968009 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:00:25.968101 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 01:00:25.968242 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260619 01:00:25.999707 7961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:26.000007 7961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:26.003948 7961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c70f922f548440e82eec8b1247a6211: Sending DeleteTablet for 3 replicas of tablet edc132ff26434ddeb0e4b15c3abc57d3
I20260619 01:00:26.004654 8100 tablet_service.cc:1514] Processing DeleteTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52370
I20260619 01:00:26.005084 8497 tablet_replica.cc:330] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: stopping tablet replica
I20260619 01:00:26.005260 8497 raft_consensus.cc:2229] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:26.005362 8497 pending_rounds.cc:62] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Trying to abort 1 pending ops.
I20260619 01:00:26.005441 8497 pending_rounds.cc:68] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7298379063210856448 op_type: PARTICIPANT_OP participant_request { tablet_id: "edc132ff26434ddeb0e4b15c3abc57d3" op { type: FINALIZE_COMMIT txn_id: 1 finalized_commit_timestamp: 7298379063171383296 } }
I20260619 01:00:26.005703 8497 raft_consensus.cc:2258] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:26.006012 8239 tablet_service.cc:1514] Processing DeleteTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43270
I20260619 01:00:26.006294 8495 tablet_replica.cc:330] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: stopping tablet replica
I20260619 01:00:26.006507 8495 raft_consensus.cc:2229] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:26.006884 8378 tablet_service.cc:1514] Processing DeleteTablet for tablet edc132ff26434ddeb0e4b15c3abc57d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:25 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47166
I20260619 01:00:26.007154 8531 tablet_replica.cc:330] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: stopping tablet replica
I20260619 01:00:26.007300 8531 raft_consensus.cc:2229] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:26.007690 8531 raft_consensus.cc:2258] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:26.007843 8495 raft_consensus.cc:2258] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:26.009334 8495 ts_tablet_manager.cc:1905] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 01:00:26.012310 8497 mvcc.cc:119] aborting op with timestamp 7298379063171383296 in state 0; MVCC is closed
I20260619 01:00:26.012387 8531 ts_tablet_manager.cc:1905] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:26.012562 8497 ts_tablet_manager.cc:1905] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:26.015837 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8034
I20260619 01:00:26.025857 8497 ts_tablet_manager.cc:1918] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:26.026054 8497 log.cc:1196] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/edc132ff26434ddeb0e4b15c3abc57d3
I20260619 01:00:26.026489 8497 ts_tablet_manager.cc:1939] T edc132ff26434ddeb0e4b15c3abc57d3 P 0d5b7badfda34f5382c6a02d481b7537: Deleting consensus metadata
I20260619 01:00:26.027753 7947 catalog_manager.cc:4841] TS 0d5b7badfda34f5382c6a02d481b7537 (127.0.97.1:38425): tablet edc132ff26434ddeb0e4b15c3abc57d3 (table test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]) successfully deleted
I20260619 01:00:26.030467 8495 ts_tablet_manager.cc:1918] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:26.033540 8531 ts_tablet_manager.cc:1918] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 01:00:26.034505 8531 log.cc:1196] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/edc132ff26434ddeb0e4b15c3abc57d3
I20260619 01:00:26.034814 8495 log.cc:1196] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/edc132ff26434ddeb0e4b15c3abc57d3
I20260619 01:00:26.035317 8531 ts_tablet_manager.cc:1939] T edc132ff26434ddeb0e4b15c3abc57d3 P 3998af71bd8249f2a5565c082539bea1: Deleting consensus metadata
I20260619 01:00:26.036181 8495 ts_tablet_manager.cc:1939] T edc132ff26434ddeb0e4b15c3abc57d3 P 1267de06ed2847b88b5a3bad45602b6a: Deleting consensus metadata
I20260619 01:00:26.037420 7950 catalog_manager.cc:4841] TS 3998af71bd8249f2a5565c082539bea1 (127.0.97.3:45293): tablet edc132ff26434ddeb0e4b15c3abc57d3 (table test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]) successfully deleted
I20260619 01:00:26.038627 7949 catalog_manager.cc:4841] TS 1267de06ed2847b88b5a3bad45602b6a (127.0.97.2:38043): tablet edc132ff26434ddeb0e4b15c3abc57d3 (table test-table [id=0d36623f887a46bdbf9d2e5e6dface8d]) successfully deleted
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260619 01:00:26.061995 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:26.016780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.0.97.1:38425)
0619 01:00:26.016957 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:26.016965 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:26.041631 (+ 24666us) server_negotiation.cc:407] Connection header received
0619 01:00:26.042031 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:26.042037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:26.042127 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:26.042213 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:26.043852 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.044915 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.046983 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.047577 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.047668 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:26.053697 (+ 6029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:26.053717 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:26.053740 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:26.053786 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:26.058066 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.059096 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.059108 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.059113 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.059274 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.059793 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.059802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.059808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.060289 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:26.060651 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:26.061264 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 01:00:26.061530 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:26.063856 8165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:26.295679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8034
I20260619 01:00:26.328876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8174
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260619 01:00:26.355021 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:26.329937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.0.97.2:38043)
0619 01:00:26.330177 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:26.330185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:26.330211 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:00:26.330473 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:26.330479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:26.330567 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:26.330688 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:26.332611 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.333743 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.335842 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.336569 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.336711 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:26.346181 (+ 9470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:26.346205 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:26.346240 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:26.346306 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:26.351071 (+ 4765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.352049 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.352061 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.352065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.352283 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.352784 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.352793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.352804 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.353361 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:26.353737 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:26.354032 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 01:00:26.354314 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:00:26.356000 8304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:26.511536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8174
W20260619 01:00:26.536245 8330 connection.cc:535] server connection from 127.0.97.2:41231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 01:00:26.536886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8313
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[7918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260619 01:00:26.553241 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:26.537685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.0.97.3:45293)
0619 01:00:26.539081 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:26.539087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:26.539106 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:26.539155 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:26.539159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:26.539227 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:26.539363 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:26.540883 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.541727 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.543213 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:26.543631 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:26.543767 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:26.548283 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:26.548298 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:26.548315 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:26.548352 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:26.551068 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.551654 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.551661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.551663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.551786 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:26.552085 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:26.552088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:26.552090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:26.552403 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:26.552625 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:26.552818 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 01:00:26.552974 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":1179,"threads_started":1}
I20260619 01:00:26.553876 8443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:26.677783 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8313
I20260619 01:00:26.699236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 7931
I20260619 01:00:26.700363 8003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:26.813730 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 7931
2026-06-19T01:00:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5657 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[8576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[8576](info): set up 2 sockets
Jun 19 01:00:26 dist-test-slave-3rxd krb5kdc[8576](info): commencing operation
krb5kdc: starting...
W20260619 01:00:27.911689 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:28 dist-test-slave-3rxd krb5kdc[8576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:00:28.122310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34735
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:28.438578 8589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:28.438915 8589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:28.439073 8589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:28.445912 8589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:28.446075 8589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:28.446159 8589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:28.446241 8589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:28.446310 8589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:28.456022 8589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:28.457226 8589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:28.458501 8589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:28.470233 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:28.470183 8597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:28.470275 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:28.471091 8589 server_base.cc:1034] running on GCE node
I20260619 01:00:28.471827 8589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:28.473080 8589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:28.474280 8589 hybrid_clock.cc:648] HybridClock initialized: now 1781830828474237 us; error 41 us; skew 500 ppm
I20260619 01:00:28.474619 8589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T01:00:28Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1724 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:28 dist-test-slave-3rxd krb5kdc[8603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:28 dist-test-slave-3rxd krb5kdc[8603](info): set up 2 sockets
Jun 19 01:00:28 dist-test-slave-3rxd krb5kdc[8603](info): commencing operation
krb5kdc: starting...
W20260619 01:00:29.623697 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:29 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:00:29.860359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36149
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:30.172227 8616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:30.172493 8616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:30.172674 8616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:30.179868 8616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:30.179991 8616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:30.180086 8616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:30.180181 8616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:30.180263 8616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:30.189915 8616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:30.191259 8616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:30.192518 8616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:30.203863 8621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:30.203984 8622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:30.203933 8624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:30.209687 8616 server_base.cc:1034] running on GCE node
I20260619 01:00:30.210409 8616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:30.211758 8616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:30.212989 8616 hybrid_clock.cc:648] HybridClock initialized: now 1781830830212948 us; error 54 us; skew 500 ppm
I20260619 01:00:30.213306 8616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:30 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:30.216245 8616 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:00:30.217659 8616 webserver.cc:460] Webserver started at http://127.0.97.62:34733/ using document root <none> and password file <none>
I20260619 01:00:30.218308 8616 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:30.218437 8616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:30.218801 8616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:30.221516 8616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "5cdb45278f9044baac7ee7be85d3718b"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "373992f74f17fdbd9f41aa252de84692"
server_key_iv: "5f7861335004fbba4ec8bbf9597fac30"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.222247 8616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "5cdb45278f9044baac7ee7be85d3718b"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "373992f74f17fdbd9f41aa252de84692"
server_key_iv: "5f7861335004fbba4ec8bbf9597fac30"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.227046 8616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 01:00:30.230132 8631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:30.231374 8616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:30.231545 8616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "5cdb45278f9044baac7ee7be85d3718b"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "373992f74f17fdbd9f41aa252de84692"
server_key_iv: "5f7861335004fbba4ec8bbf9597fac30"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.231734 8616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:30.252594 8616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:30.256822 8616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:30.257093 8616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:30.275159 8616 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36149
I20260619 01:00:30.275210 8682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36149 every 8 connection(s)
I20260619 01:00:30.276485 8616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:30.279534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8616
I20260619 01:00:30.279732 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:30.280567 388 external_mini_cluster.cc:1428] Setting key 1d13b8dd653dd797b56b800f07c26cb8
I20260619 01:00:30.280565 8683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:30.295342 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Bootstrap starting.
Jun 19 01:00:30 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:30.298487 8683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:30.299607 8683 log.cc:824] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:30.302474 8683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: No bootstrap required, opened a new log
I20260619 01:00:30.303686 8686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:30.282614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42304 (local address 127.0.97.62:36149)
0619 01:00:30.283482 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:30.283494 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:30.283535 (+ 41us) server_negotiation.cc:407] Connection header received
0619 01:00:30.285327 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:30.285349 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:30.285767 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:30.286102 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:30.287994 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.289285 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.291064 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.291791 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.291944 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:30.296496 (+ 4552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:30.296504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:30.296519 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:30.296559 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:30.299658 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:30.300360 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:30.300366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:30.300373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:30.300527 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:30.300963 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:30.300971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:30.300976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:30.301479 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:30.301784 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:30.302211 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 01:00:30.302664 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":369,"threads_started":1}
I20260619 01:00:30.309971 8683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:30.310307 8683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:30.310425 8683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb45278f9044baac7ee7be85d3718b, State: Initialized, Role: FOLLOWER
I20260619 01:00:30.311050 8683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:30.311319 8683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:30.311482 8683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:30.311683 8683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:30.313707 8683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:30.314117 8683 leader_election.cc:302] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cdb45278f9044baac7ee7be85d3718b; no voters:
I20260619 01:00:30.314579 8683 leader_election.cc:288] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:30.314718 8688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:30.315388 8688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 LEADER]: Becoming Leader. State: Replica: 5cdb45278f9044baac7ee7be85d3718b, State: Running, Role: LEADER
I20260619 01:00:30.315889 8688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:30.316669 8683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:30.318641 8690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cdb45278f9044baac7ee7be85d3718b. Latest consensus state: current_term: 1 leader_uuid: "5cdb45278f9044baac7ee7be85d3718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } } }
I20260619 01:00:30.319041 8690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:30.319562 8689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cdb45278f9044baac7ee7be85d3718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } } }
I20260619 01:00:30.320014 8689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:30.321357 8696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:30.326316 8696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:30.335786 8696 catalog_manager.cc:1347] Generated new cluster ID: d67c50bb95884a1182444bac3bb02a1b
I20260619 01:00:30.335947 8696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:30.349278 8696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:30.350425 8696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:30.360060 8696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Generated new TSK 0
I20260619 01:00:30.360816 8696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:00:30.422091 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:30.737253 8709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:30.737509 8709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:30.737679 8709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:30.744608 8709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:30.744726 8709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:30.744905 8709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:30.754138 8709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:30.755383 8709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:30.756618 8709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:30.769111 8714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:30.769184 8715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:30.769155 8717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:30.770299 8709 server_base.cc:1034] running on GCE node
I20260619 01:00:30.770984 8709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:30.771850 8709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:30.773072 8709 hybrid_clock.cc:648] HybridClock initialized: now 1781830830773029 us; error 55 us; skew 500 ppm
I20260619 01:00:30.773366 8709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:30 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:30.776364 8709 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:00:30.777689 8709 webserver.cc:460] Webserver started at http://127.0.97.1:38155/ using document root <none> and password file <none>
I20260619 01:00:30.778331 8709 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:30.778471 8709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:30.778841 8709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:30.781553 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "77f60e851dbf4f2d86d50d01e378787e"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "4167ccef15a785c3a8d5aa300ee9af31"
server_key_iv: "2972874d6098efaca41b0c208f15305f"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.782306 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "77f60e851dbf4f2d86d50d01e378787e"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "4167ccef15a785c3a8d5aa300ee9af31"
server_key_iv: "2972874d6098efaca41b0c208f15305f"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.786964 8709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 01:00:30.790218 8724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:30.791494 8709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:30.791672 8709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "77f60e851dbf4f2d86d50d01e378787e"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "4167ccef15a785c3a8d5aa300ee9af31"
server_key_iv: "2972874d6098efaca41b0c208f15305f"
server_key_version: "encryptionkey@0"
I20260619 01:00:30.791860 8709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:30.804416 8709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:30.808066 8709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:30.808342 8709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:30.809185 8709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:30.810417 8709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:30.810520 8709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:30.810672 8709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:30.810781 8709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 01:00:30.842834 8709 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34209
I20260619 01:00:30.842852 8836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34209 every 8 connection(s)
I20260619 01:00:30.844127 8709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:30.852713 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8709
I20260619 01:00:30.852911 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:30.853538 388 external_mini_cluster.cc:1428] Setting key 6b4de6c53f8dafe982ff801a24c3851b
Jun 19 01:00:30 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:30.865272 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:30.846553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47077 (local address 127.0.97.62:36149)
0619 01:00:30.847002 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:30.847007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:30.848035 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 01:00:30.849843 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:30.849849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:30.849938 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:30.850055 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:30.851589 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.852384 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.854095 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.854623 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.854781 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:30.858816 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:30.858821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:30.858824 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:30.858854 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:30.862142 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:30.863028 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:30.863033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:30.863035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:30.863161 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:30.863588 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:30.863595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:30.863602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:30.863862 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:30.864010 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:30.864687 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 01:00:30.864896 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":352,"thread_start_us":245,"threads_started":1}
I20260619 01:00:30.866243 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:30.847188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:47077)
0619 01:00:30.847872 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:30.847908 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:30.849549 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:30.850296 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:30.850305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:30.850841 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:30.851416 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:30.851432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.852531 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.852534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:30.853942 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:30.853954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:30.854867 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:30.854875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:30.855125 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:30.855876 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:30.855898 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:30.858541 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:30.862290 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:30.862302 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:30.862323 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:30.862894 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:30.863272 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:30.863278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:30.863282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:30.863464 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:30.863985 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:30.863993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:30.864333 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 01:00:30.864595 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 01:00:30.865107 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260619 01:00:30.867790 8837 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36149
I20260619 01:00:30.868111 8837 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:30.868932 8837 heartbeater.cc:511] Master 127.0.97.62:36149 requested a full tablet report, sending...
I20260619 01:00:30.871101 8648 ts_manager.cc:194] Registered new tserver with Master: 77f60e851dbf4f2d86d50d01e378787e (127.0.97.1:34209)
I20260619 01:00:30.872622 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47077
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:00:30.903956 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:31.214023 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:31.214318 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:31.214483 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:31.221556 8843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:31.221689 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:31.221851 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:31.230901 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:31.232064 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:31.233309 8843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:31.245566 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:31.245627 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:31.245592 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:31.246471 8843 server_base.cc:1034] running on GCE node
I20260619 01:00:31.247159 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:31.247953 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:31.249169 8843 hybrid_clock.cc:648] HybridClock initialized: now 1781830831249140 us; error 44 us; skew 500 ppm
I20260619 01:00:31.249449 8843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:31 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:31.252377 8843 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:00:31.253711 8843 webserver.cc:460] Webserver started at http://127.0.97.2:37997/ using document root <none> and password file <none>
I20260619 01:00:31.254343 8843 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:31.254483 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:31.254850 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:31.257519 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "c73e9de8774e4251a7b06dc0b54c1967"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "4fdefe77f5ea300d0be403ebbe5f7603"
server_key_iv: "c2f8ccdc172625fe17c9f3044c1bbd0e"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.258262 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c73e9de8774e4251a7b06dc0b54c1967"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "4fdefe77f5ea300d0be403ebbe5f7603"
server_key_iv: "c2f8ccdc172625fe17c9f3044c1bbd0e"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.263007 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260619 01:00:31.266114 8858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.267376 8843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:31.267544 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "c73e9de8774e4251a7b06dc0b54c1967"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "4fdefe77f5ea300d0be403ebbe5f7603"
server_key_iv: "c2f8ccdc172625fe17c9f3044c1bbd0e"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.267731 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:31.288321 8843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:31.292407 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:31.292688 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:31.293524 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:31.294790 8843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:31.294886 8843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.295006 8843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:31.295083 8843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.327586 8843 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41841
I20260619 01:00:31.327632 8970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41841 every 8 connection(s)
I20260619 01:00:31.328953 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:31.332966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8843
I20260619 01:00:31.333169 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:31.333632 388 external_mini_cluster.cc:1428] Setting key 65f4d45ddfc01a2721ce29c194755c29
Jun 19 01:00:31 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:00:31.354605 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:31.331537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35503 (local address 127.0.97.62:36149)
0619 01:00:31.331749 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:31.331756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:31.333018 (+ 1262us) server_negotiation.cc:407] Connection header received
0619 01:00:31.335087 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:31.335092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:31.335178 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:31.335295 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:31.336784 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.337942 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.339494 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.340224 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.340359 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:31.348937 (+ 8578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:31.348942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:31.348946 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:31.348976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:31.351581 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.352437 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.352441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.352443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.352563 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.353019 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.353022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.353024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.353248 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:31.353345 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:31.354203 (+ 858us) server_negotiation.cc:299] Negotiation successful
0619 01:00:31.354380 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:00:31.355707 8973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:31.332124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:35503)
0619 01:00:31.332833 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:31.332872 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:31.334844 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:31.335591 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:31.335604 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:31.336130 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:31.336629 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:31.336644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.338078 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.338081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:31.339347 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:31.339358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.340409 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.340413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:31.340585 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:31.346460 (+ 5875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:31.346486 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:31.348731 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:31.351715 (+ 2984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:31.351726 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:31.351744 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:31.352321 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:31.352683 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:31.352688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:31.352692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:31.352914 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:31.353368 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:31.353378 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:31.353815 (+ 437us) client_negotiation.cc:769] Sending connection context
0619 01:00:31.354100 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 01:00:31.354576 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":274,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:00:31.357111 8971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36149
I20260619 01:00:31.357411 8971 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:31.358067 8971 heartbeater.cc:511] Master 127.0.97.62:36149 requested a full tablet report, sending...
I20260619 01:00:31.359395 8648 ts_manager.cc:194] Registered new tserver with Master: c73e9de8774e4251a7b06dc0b54c1967 (127.0.97.2:41841)
I20260619 01:00:31.360177 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:00:31.381803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:31.702943 8976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:31.703225 8976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:31.703482 8976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:31.710603 8976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:31.710804 8976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:31.711009 8976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:31.720875 8976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:31.722328 8976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:31.723650 8976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:31.736367 8982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:31.736416 8984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:31.736330 8981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:31.737113 8976 server_base.cc:1034] running on GCE node
I20260619 01:00:31.737723 8976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:31.738687 8976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:31.739974 8976 hybrid_clock.cc:648] HybridClock initialized: now 1781830831739908 us; error 49 us; skew 500 ppm
I20260619 01:00:31.740314 8976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:31 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:31.743443 8976 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:00:31.744863 8976 webserver.cc:460] Webserver started at http://127.0.97.3:45319/ using document root <none> and password file <none>
I20260619 01:00:31.745518 8976 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:31.745663 8976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:31.745965 8976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:31.748879 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "e15b4072d74d478aaed4855239501f45"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "ccd52affe325beb3d1d03f40357869b5"
server_key_iv: "ff0d87bffe7bf863beaa6ae687d225d3"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.749629 8976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e15b4072d74d478aaed4855239501f45"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "ccd52affe325beb3d1d03f40357869b5"
server_key_iv: "ff0d87bffe7bf863beaa6ae687d225d3"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.754498 8976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 01:00:31.757746 8991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.759188 8976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:31.759366 8976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "e15b4072d74d478aaed4855239501f45"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "ccd52affe325beb3d1d03f40357869b5"
server_key_iv: "ff0d87bffe7bf863beaa6ae687d225d3"
server_key_version: "encryptionkey@0"
I20260619 01:00:31.759564 8976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:31.777382 8976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:31.782078 8976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:31.782375 8976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:31.783397 8976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:31.784739 8976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:31.784842 8976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.784996 8976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:31.785069 8976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:31.818096 8976 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45891
I20260619 01:00:31.818140 9103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45891 every 8 connection(s)
I20260619 01:00:31.819417 8976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:31.820014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 8976
I20260619 01:00:31.820190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:31.820811 388 external_mini_cluster.cc:1428] Setting key e6ff00d5c90f9499fbfa156a1f52439f
Jun 19 01:00:31 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:31.840044 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:31.822454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45377 (local address 127.0.97.62:36149)
0619 01:00:31.822627 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:31.822631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:31.824352 (+ 1721us) server_negotiation.cc:407] Connection header received
0619 01:00:31.826384 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:31.826388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:31.826451 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:31.826534 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:31.828049 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.828771 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.830411 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.830893 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.830988 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:31.834129 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:31.834134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:31.834137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:31.834167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:31.837197 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.837877 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.837884 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.837890 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.838011 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.838412 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.838419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.838422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.838691 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:31.838868 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:31.839503 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 01:00:31.839714 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:31.841064 9106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:31.823194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:45377)
0619 01:00:31.824196 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:31.824244 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:31.826136 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:31.826820 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:31.826829 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:31.827357 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:31.827907 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:31.827923 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.828897 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.828901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:31.830290 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:31.830302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.831078 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.831083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:31.831286 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:31.831879 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:31.831900 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:31.833950 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:31.837332 (+ 3382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:31.837341 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:31.837358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:31.837762 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:31.838124 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:31.838129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:31.838133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:31.838307 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:31.838839 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:31.838847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:31.839167 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 01:00:31.839425 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 01:00:31.839909 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":378,"threads_started":1}
I20260619 01:00:31.842408 9104 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36149
I20260619 01:00:31.842734 9104 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:31.843675 9104 heartbeater.cc:511] Master 127.0.97.62:36149 requested a full tablet report, sending...
I20260619 01:00:31.845142 8648 ts_manager.cc:194] Registered new tserver with Master: e15b4072d74d478aaed4855239501f45 (127.0.97.3:45891)
I20260619 01:00:31.845917 8648 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45377
I20260619 01:00:31.848508 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
I20260619 01:00:31.876128 8837 heartbeater.cc:503] Master 127.0.97.62:36149 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260619 01:00:31.974932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8709
Jun 19 01:00:31 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:31.990181 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:31.975993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.0.97.1:34209)
0619 01:00:31.976617 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:31.976631 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:31.976658 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:31.976716 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:31.976722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:31.976932 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:31.977063 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:31.978353 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.979418 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.980648 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:31.981187 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:31.981324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:31.985054 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:31.985065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:31.985079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:31.985111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:31.987727 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.988356 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.988362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.988365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.988482 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:31.988779 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:31.988783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:31.988784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:31.989078 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:31.989283 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:31.989579 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 01:00:31.989778 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":324,"threads_started":1}
I20260619 01:00:31.991447 8832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:32.077049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8709
I20260619 01:00:32.093840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8843
Jun 19 01:00:32 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:00:32.109797 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:32.094776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51168 (local address 127.0.97.2:41841)
0619 01:00:32.095404 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:32.095411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:32.095436 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:00:32.095494 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:32.095500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:32.095721 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:32.095850 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:32.097068 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:32.098071 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:32.099477 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:32.100023 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:32.100118 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:32.104136 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:32.104149 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:32.104163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:32.104203 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:32.107308 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:32.107905 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:32.107912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:32.107915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:32.108072 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:32.108360 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:32.108363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:32.108365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:32.108703 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:32.108942 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:32.109160 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 01:00:32.109419 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":337,"threads_started":1}
I20260619 01:00:32.111086 8966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:32.193048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8843
I20260619 01:00:32.208793 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8976
Jun 19 01:00:32 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:00:32.224202 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:32.209804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.0.97.3:45891)
0619 01:00:32.210426 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:32.210433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:32.210460 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:32.210521 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:32.210527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:32.210788 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:32.210920 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:32.212094 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:32.213057 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:32.214197 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:32.214787 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:32.214883 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:32.219202 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:32.219212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:32.219226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:32.219254 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:32.221886 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:32.222500 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:32.222506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:32.222508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:32.222623 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:32.222955 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:32.222958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:32.222960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:32.223256 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:32.223491 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:32.223653 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 01:00:32.223833 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":352,"threads_started":1}
I20260619 01:00:32.225379 9099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:32.305852 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8976
I20260619 01:00:32.322194 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 8616
I20260619 01:00:32.323652 8678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:32.425272 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 8616
I20260619 01:00:32.445118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36149
--webserver_interface=127.0.97.62
--webserver_port=34733
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:32.757395 9121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:32.757655 9121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:32.757742 9121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:32.764745 9121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:32.764863 9121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:32.764909 9121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:32.764968 9121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:32.765012 9121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:32.774462 9121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:32.775764 9121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:32.776985 9121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:32.788132 9126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:32.788172 9127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:32.788290 9129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:32.789134 9121 server_base.cc:1034] running on GCE node
I20260619 01:00:32.789893 9121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:32.791224 9121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:32.792434 9121 hybrid_clock.cc:648] HybridClock initialized: now 1781830832792395 us; error 55 us; skew 500 ppm
I20260619 01:00:32.792768 9121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:32 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:32.795763 9121 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:00:32.797139 9121 webserver.cc:460] Webserver started at http://127.0.97.62:34733/ using document root <none> and password file <none>
I20260619 01:00:32.797711 9121 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:32.797816 9121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:32.802793 9121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:00:32.805279 9136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:32.806519 9121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:00:32.806650 9121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "5cdb45278f9044baac7ee7be85d3718b"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "373992f74f17fdbd9f41aa252de84692"
server_key_iv: "5f7861335004fbba4ec8bbf9597fac30"
server_key_version: "encryptionkey@0"
I20260619 01:00:32.807554 9121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:32.820488 9121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:32.824074 9121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:32.824267 9121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:32.841603 9121 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36149
I20260619 01:00:32.841645 9187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36149 every 8 connection(s)
I20260619 01:00:32.842995 9121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:32.843818 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9121
I20260619 01:00:32.844762 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:34209
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:36149
--webserver_port=38155
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:00:32.852438 9188 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:00:32.855561 9188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Bootstrap starting.
I20260619 01:00:32.873467 9188 log.cc:824] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:32.881373 9188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:00:32.881870 9188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Bootstrap complete.
I20260619 01:00:32.890462 9188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:32.891361 9188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb45278f9044baac7ee7be85d3718b, State: Initialized, Role: FOLLOWER
I20260619 01:00:32.892045 9188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:32.892266 9188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:32.892369 9188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:32.892519 9188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:00:32.896169 9188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:32.896660 9188 leader_election.cc:302] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cdb45278f9044baac7ee7be85d3718b; no voters:
I20260619 01:00:32.897305 9188 leader_election.cc:288] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:00:32.897428 9192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:00:32.898589 9192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [term 2 LEADER]: Becoming Leader. State: Replica: 5cdb45278f9044baac7ee7be85d3718b, State: Running, Role: LEADER
I20260619 01:00:32.899191 9192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } }
I20260619 01:00:32.899468 9188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:32.901830 9193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5cdb45278f9044baac7ee7be85d3718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } } }
I20260619 01:00:32.902118 9193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:32.901465 9194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cdb45278f9044baac7ee7be85d3718b. Latest consensus state: current_term: 2 leader_uuid: "5cdb45278f9044baac7ee7be85d3718b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb45278f9044baac7ee7be85d3718b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36149 } } }
I20260619 01:00:32.902649 9194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:32.912323 9198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:32.918015 9198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:32.919346 9198 catalog_manager.cc:1259] Loaded cluster ID: d67c50bb95884a1182444bac3bb02a1b
I20260619 01:00:32.919492 9198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:32.923408 9198 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:32.924789 9198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5cdb45278f9044baac7ee7be85d3718b: Loaded TSK: 0
I20260619 01:00:32.926294 9198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:00:33.178256 9190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:33.178534 9190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:33.178671 9190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:33.187487 9190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:33.187592 9190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:33.187768 9190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:33.197417 9190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:34209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=38155
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:33.198764 9190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:33.200145 9190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:33.213828 9214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:33.213959 9215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:33.213929 9217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:33.214929 9190 server_base.cc:1034] running on GCE node
I20260619 01:00:33.215482 9190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:33.216352 9190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:33.217574 9190 hybrid_clock.cc:648] HybridClock initialized: now 1781830833217531 us; error 41 us; skew 500 ppm
I20260619 01:00:33.217869 9190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:33.220950 9190 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260619 01:00:33.222440 9190 webserver.cc:460] Webserver started at http://127.0.97.1:38155/ using document root <none> and password file <none>
I20260619 01:00:33.223146 9190 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:33.223281 9190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:33.228302 9190 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 01:00:33.230970 9224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.232383 9190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:33.232548 9190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "77f60e851dbf4f2d86d50d01e378787e"
format_stamp: "Formatted at 2026-06-19 01:00:30 on dist-test-slave-3rxd"
server_key: "4167ccef15a785c3a8d5aa300ee9af31"
server_key_iv: "2972874d6098efaca41b0c208f15305f"
server_key_version: "encryptionkey@0"
I20260619 01:00:33.233439 9190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:33.252043 9190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:33.256222 9190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:33.256484 9190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:33.257404 9190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:33.258602 9190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:33.258764 9190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.258889 9190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:33.258975 9190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.291734 9190 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34209
I20260619 01:00:33.291821 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34209 every 8 connection(s)
I20260619 01:00:33.293138 9190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:33.297501 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9190
I20260619 01:00:33.298099 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41841
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:36149
--webserver_port=37997
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:33.311389 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.296140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:45733)
0619 01:00:33.297254 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.297288 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.299096 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.300561 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.300573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.301174 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.301672 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.301688 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.303381 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.303385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.304540 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.304551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.305613 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.305620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.305894 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.306542 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.306561 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.310169 (+ 3608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":756,"threads_started":1}
W20260619 01:00:33.311734 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.296046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45733 (local address 127.0.97.62:36149)
0619 01:00:33.296805 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.296817 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.297416 (+ 599us) server_negotiation.cc:407] Connection header received
0619 01:00:33.299745 (+ 2329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.299757 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.300066 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.300441 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.301886 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.303217 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.304738 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.305394 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.305585 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.310514 (+ 4929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45733: BlockingRecv error: recv got EOF from 127.0.97.1:45733 (error 108)
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":332,"threads_started":1}
W20260619 01:00:33.312173 9337 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36149 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36149: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:33.321070 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.312801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:49957)
0619 01:00:33.313090 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.313105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.313186 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.313528 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.313530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.313856 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.313975 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.313982 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.315502 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.315506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.316349 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.316358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.317394 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.317398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.317511 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.318101 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.318117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.320890 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182}
W20260619 01:00:33.321452 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.312933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49957 (local address 127.0.97.62:36149)
0619 01:00:33.313127 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.313136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.313163 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:33.313298 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.313305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.313390 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.313498 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.314131 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.315367 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.316486 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.317259 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.317406 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.321114 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49957: BlockingRecv error: recv got EOF from 127.0.97.1:49957 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:33.328992 9339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.321723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:48931)
0619 01:00:33.321873 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.321891 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.321984 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.322490 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.322494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.322838 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.322966 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.322974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.323881 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.323887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.324752 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.324762 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.325427 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.325432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.325588 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.326134 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.326149 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.328713 (+ 2564us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:00:33.329136 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.322121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48931 (local address 127.0.97.62:36149)
0619 01:00:33.322257 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.322261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.322274 (+ 13us) server_negotiation.cc:407] Connection header received
0619 01:00:33.322308 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.322311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.322369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.322470 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.323091 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.323753 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.324885 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.325289 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.325419 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.328879 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48931: BlockingRecv error: recv got EOF from 127.0.97.1:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 01:00:33.329275 9337 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:00:33.624958 9341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:33.625252 9341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:33.625345 9341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:33.632776 9341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:33.632908 9341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:33.633195 9341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:33.643350 9341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37997
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:33.644624 9341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:33.645869 9341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:33.658900 9346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:33.658946 9349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:33.658977 9347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:33.659888 9341 server_base.cc:1034] running on GCE node
I20260619 01:00:33.660456 9341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:33.661301 9341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:33.662503 9341 hybrid_clock.cc:648] HybridClock initialized: now 1781830833662462 us; error 37 us; skew 500 ppm
I20260619 01:00:33.662875 9341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830833, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:33.665978 9341 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260619 01:00:33.667456 9341 webserver.cc:460] Webserver started at http://127.0.97.2:37997/ using document root <none> and password file <none>
I20260619 01:00:33.668134 9341 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:33.668288 9341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:33.673429 9341 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 01:00:33.675943 9356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.677228 9341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:33.677388 9341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "c73e9de8774e4251a7b06dc0b54c1967"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "4fdefe77f5ea300d0be403ebbe5f7603"
server_key_iv: "c2f8ccdc172625fe17c9f3044c1bbd0e"
server_key_version: "encryptionkey@0"
I20260619 01:00:33.678319 9341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:33.712967 9341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:33.717325 9341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:33.717617 9341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:33.718595 9341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:33.719964 9341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:33.720088 9341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.720229 9341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:33.720314 9341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:33.753492 9341 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41841
I20260619 01:00:33.753537 9468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41841 every 8 connection(s)
I20260619 01:00:33.754899 9341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:33.758795 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9341
I20260619 01:00:33.759588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45891
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:36149
--webserver_port=45319
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:00:33.773313 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.757500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41293 (local address 127.0.97.62:36149)
0619 01:00:33.757717 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.757723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.759107 (+ 1384us) server_negotiation.cc:407] Connection header received
0619 01:00:33.761954 (+ 2847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.761959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.762021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.762095 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.764076 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.764769 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.767188 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.767598 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.767729 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.773135 (+ 5406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41293: BlockingRecv error: recv got EOF from 127.0.97.2:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:00:33.774255 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.757997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:41293)
0619 01:00:33.758933 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.758989 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.761650 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.762351 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.762364 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.763135 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.763902 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.763927 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.764931 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.764939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.767029 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.767042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.767814 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.767822 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.768044 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.768710 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.768730 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.772987 (+ 4257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":361,"threads_started":1}
W20260619 01:00:33.774956 9469 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36149 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36149: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:00:33.783979 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.775605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55091 (local address 127.0.97.62:36149)
0619 01:00:33.775791 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.775799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.775836 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:00:33.775886 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.775893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.775969 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.776063 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.776630 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.777830 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.779035 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.779801 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.779952 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.783828 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55091: BlockingRecv error: recv got EOF from 127.0.97.2:55091 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:00:33.784029 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.775517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:55091)
0619 01:00:33.775684 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.775700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.775770 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.776088 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.776091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.776347 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.776463 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.776469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.777993 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.777997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.778905 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.778915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.779939 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.779943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.780058 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.780609 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.780623 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.783690 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:33 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:33.792198 9471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:33.784794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:60337)
0619 01:00:33.784953 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:33.784980 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:33.785055 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:33.785305 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:33.785308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:33.785561 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:33.785687 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.785694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.786992 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.786995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.787835 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:33.787845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.788658 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.788662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:33.788770 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.789277 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:33.789292 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:33.791921 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:00:33.792205 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:33.784898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60337 (local address 127.0.97.62:36149)
0619 01:00:33.785050 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:33.785056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:33.785073 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:00:33.785120 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:33.785125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:33.785193 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:33.785281 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:33.785799 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.786877 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.787950 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:33.788543 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:33.788651 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:33.792073 (+ 3422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60337: BlockingRecv error: recv got EOF from 127.0.97.2:60337 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 01:00:33.792583 9469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:00:34.085384 9472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:34.085676 9472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:34.085837 9472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:34.092854 9472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:34.092998 9472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:34.093165 9472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:34.102432 9472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45319
--tserver_master_addrs=127.0.97.62:36149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:34.103705 9472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:34.104933 9472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:34.117781 9480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:34.117866 9477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:34.117846 9478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:34.118592 9472 server_base.cc:1034] running on GCE node
I20260619 01:00:34.119197 9472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:34.120033 9472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:34.121253 9472 hybrid_clock.cc:648] HybridClock initialized: now 1781830834121224 us; error 42 us; skew 500 ppm
I20260619 01:00:34.121536 9472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830834, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:34.124406 9472 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260619 01:00:34.125694 9472 webserver.cc:460] Webserver started at http://127.0.97.3:45319/ using document root <none> and password file <none>
I20260619 01:00:34.126331 9472 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:34.126478 9472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:34.131546 9472 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:34.134061 9487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:34.135434 9472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:34.135596 9472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "e15b4072d74d478aaed4855239501f45"
format_stamp: "Formatted at 2026-06-19 01:00:31 on dist-test-slave-3rxd"
server_key: "ccd52affe325beb3d1d03f40357869b5"
server_key_iv: "ff0d87bffe7bf863beaa6ae687d225d3"
server_key_version: "encryptionkey@0"
I20260619 01:00:34.136536 9472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:34.150306 9472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:34.154016 9472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:34.154296 9472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:34.155267 9472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:34.156453 9472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:34.156571 9472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:34.156706 9472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:34.156797 9472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:34.189530 9472 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45891
I20260619 01:00:34.189584 9599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45891 every 8 connection(s)
I20260619 01:00:34.191036 9472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:34.193001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9472
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 01:00:34.209278 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:34.193657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34367 (local address 127.0.97.62:36149)
0619 01:00:34.194281 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.194291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.195459 (+ 1168us) server_negotiation.cc:407] Connection header received
0619 01:00:34.198861 (+ 3402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.198869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.198979 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.199085 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.200744 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.201494 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.203109 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.203588 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.203722 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.209060 (+ 5338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34367: BlockingRecv error: recv got EOF from 127.0.97.3:34367 (error 108)
Metrics: {"server-negotiator.queue_time_us":457,"spinlock_wait_cycles":28544}
I20260619 01:00:34.210258 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.194188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:34367)
0619 01:00:34.195248 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:34.195309 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:34.198493 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:34.199296 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:34.199304 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:34.199934 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:34.200482 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.200526 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.201644 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.201650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.202932 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.202944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.203818 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.203827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.204033 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.204612 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:34.204634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:34.208859 (+ 4225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":415,"threads_started":1}
W20260619 01:00:34.210994 9600 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:36149 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:36149: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260619 01:00:34.213397 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.193982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35272 (local address 127.0.97.62:36149)
0619 01:00:34.199082 (+ 5100us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.199087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.199101 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:00:34.199140 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.199143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.199225 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.199348 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.200877 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.201615 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.202844 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.203269 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.203410 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.206016 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:34.206046 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:34.206066 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:34.206115 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:34.209977 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:34.210848 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:34.210856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:34.210863 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:34.211052 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:34.211476 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:34.211484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:34.211490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:34.211992 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:34.212303 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:34.212702 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 01:00:34.212941 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4989,"spinlock_wait_cycles":2432,"thread_start_us":4878,"threads_started":1}
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:34.219951 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.211640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:54823)
0619 01:00:34.211796 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:34.211815 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:34.211929 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:34.212357 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:34.212362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:34.212753 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:34.212929 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.212939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.214196 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.214201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.215588 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.215599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.216566 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.216572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.216755 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.217275 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:34.217290 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:34.219769 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 01:00:34.220118 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:34.211725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54823 (local address 127.0.97.62:36149)
0619 01:00:34.211878 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.211883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.211898 (+ 15us) server_negotiation.cc:407] Connection header received
0619 01:00:34.212025 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.212030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.212094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.212189 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.213117 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.214060 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.216035 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.216425 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.216556 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.219931 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54823: BlockingRecv error: recv got EOF from 127.0.97.3:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:34.227429 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.220598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:58333)
0619 01:00:34.220740 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:34.220756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:34.220867 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:34.221162 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:34.221167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:34.221439 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:34.221564 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.221572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.222512 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.222517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.223459 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.223516 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.224160 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.224166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.224325 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.224753 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:34.224767 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:34.227181 (+ 2414us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:00:34.227722 9600 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:00:34.227952 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:34.220675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58333 (local address 127.0.97.62:36149)
0619 01:00:34.220897 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.220905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.220924 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:00:34.220975 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.220978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.221024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.221121 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.221692 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.222384 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.223656 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.224016 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.224144 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.227355 (+ 3211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58333: BlockingRecv error: recv got EOF from 127.0.97.3:58333 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:34.338277 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.329890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:36419)
0619 01:00:34.330418 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:34.330434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:34.330544 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:34.330907 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:34.330910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:34.331351 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:34.331541 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.331549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.332481 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.332484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.333409 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.333418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.334100 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.334104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.334267 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.335044 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:34.335071 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:34.338059 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":287,"threads_started":1}
W20260619 01:00:34.338420 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:34.329943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36419 (local address 127.0.97.62:36149)
0619 01:00:34.330111 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.330115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.330511 (+ 396us) server_negotiation.cc:407] Connection header received
0619 01:00:34.330650 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.330654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.330761 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.330858 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.331701 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.332365 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.333520 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.333946 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.334056 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.338238 (+ 4182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36419: BlockingRecv error: recv got EOF from 127.0.97.1:36419 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:34 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:34.801383 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:34.793103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:35187)
0619 01:00:34.793644 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:34.793660 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:34.793771 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:34.794042 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:34.794045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:34.794485 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:34.794655 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.794663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.795659 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.795664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.796927 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:34.796943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.797633 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.797647 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:34.797807 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.798338 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:34.798353 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:34.801166 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":324,"threads_started":1}
W20260619 01:00:34.801577 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:34.793204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35187 (local address 127.0.97.62:36149)
0619 01:00:34.793431 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:34.793435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:34.793731 (+ 296us) server_negotiation.cc:407] Connection header received
0619 01:00:34.793864 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:34.793868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:34.793924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:34.794001 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:34.794791 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.795493 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.797050 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:34.797459 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:34.797600 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:34.801341 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35187: BlockingRecv error: recv got EOF from 127.0.97.2:35187 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:35.236590 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:35.228393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:52449)
0619 01:00:35.228880 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:35.228908 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:35.229050 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:35.229390 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:35.229394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:35.229850 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:35.230030 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.230038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.231116 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.231122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.232059 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.232070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.232772 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.232779 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.232946 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.233481 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:35.233497 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:35.236276 (+ 2779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":263,"threads_started":1}
W20260619 01:00:35.236730 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:35.228432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52449 (local address 127.0.97.62:36149)
0619 01:00:35.228616 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:35.228625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:35.228992 (+ 367us) server_negotiation.cc:407] Connection header received
0619 01:00:35.229190 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:35.229196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:35.229259 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:35.229358 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:35.230183 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.230971 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.232197 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.232584 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.232713 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.236505 (+ 3792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:52449: BlockingRecv error: recv got EOF from 127.0.97.3:52449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:35.347676 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:35.339272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:44965)
0619 01:00:35.339723 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:35.339739 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:35.339918 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:35.340289 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:35.340294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:35.340694 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:35.340827 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.340835 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.341849 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.341856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.342808 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.342820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.343503 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.343509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.343668 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.344218 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:35.344234 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:35.347339 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
W20260619 01:00:35.347771 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:35.339347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44965 (local address 127.0.97.62:36149)
0619 01:00:35.339563 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:35.339570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:35.339854 (+ 284us) server_negotiation.cc:407] Connection header received
0619 01:00:35.340040 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:35.340046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:35.340146 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:35.340261 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:35.340963 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.341711 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.342936 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.343342 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.343489 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.347521 (+ 4032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44965: BlockingRecv error: recv got EOF from 127.0.97.1:44965 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:35 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:35.810735 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:35.802245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:36849)
0619 01:00:35.802811 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:35.802828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:35.802940 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:35.803212 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:35.803215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:35.803594 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:35.803753 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.803762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.804707 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.804710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.805583 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:35.805593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.806492 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.806499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:35.806713 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.807498 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:35.807523 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:35.810353 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":295,"threads_started":1}
W20260619 01:00:35.810755 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:35.802281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36849 (local address 127.0.97.62:36149)
0619 01:00:35.802457 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:35.802461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:35.802895 (+ 434us) server_negotiation.cc:407] Connection header received
0619 01:00:35.803031 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:35.803035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:35.803092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:35.803184 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:35.803868 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.804603 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.805727 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:35.806266 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:35.806388 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:35.810535 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36849: BlockingRecv error: recv got EOF from 127.0.97.2:36849 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:36.245834 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:36.237418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:45339)
0619 01:00:36.238000 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:36.238019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:36.238124 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:36.238465 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:36.238471 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:36.239011 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:36.239158 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.239166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.240175 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.240178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.241299 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.241311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.241993 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.241998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.242104 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.242881 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:36.242896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:36.245594 (+ 2698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":349,"threads_started":1}
W20260619 01:00:36.246086 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:36.237615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45339 (local address 127.0.97.62:36149)
0619 01:00:36.237813 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:36.237818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:36.238096 (+ 278us) server_negotiation.cc:407] Connection header received
0619 01:00:36.238221 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:36.238225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:36.238294 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:36.238381 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:36.239324 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.240044 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.241415 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.241831 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.241945 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.245805 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45339: BlockingRecv error: recv got EOF from 127.0.97.3:45339 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:36.365159 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:36.348562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:39319)
0619 01:00:36.349088 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:36.349105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:36.349226 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:36.349573 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:36.349577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:36.349966 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:36.350124 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.350132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.351217 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.351225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.352111 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.352122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.352796 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.352801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.352973 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.362128 (+ 9155us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:36.362147 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:36.364907 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":306,"threads_started":1}
W20260619 01:00:36.365326 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:36.348603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39319 (local address 127.0.97.62:36149)
0619 01:00:36.348808 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:36.348814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:36.349177 (+ 363us) server_negotiation.cc:407] Connection header received
0619 01:00:36.349356 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:36.349363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:36.349435 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:36.349542 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:36.350271 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.351075 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.352251 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.352643 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.352783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.365094 (+ 12311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:39319: BlockingRecv error: recv got EOF from 127.0.97.1:39319 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:36 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:36.820307 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:36.811815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:53989)
0619 01:00:36.812303 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:36.812318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:36.812466 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:36.812786 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:36.812790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:36.813173 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:36.813338 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.813347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.814423 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.814432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.815357 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:36.815369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.816033 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.816039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:36.816212 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.816916 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:36.816934 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:36.819965 (+ 3031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":268,"threads_started":1}
W20260619 01:00:36.820469 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:36.811884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53989 (local address 127.0.97.62:36149)
0619 01:00:36.812137 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:36.812147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:36.812396 (+ 249us) server_negotiation.cc:407] Connection header received
0619 01:00:36.812589 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:36.812595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:36.812666 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:36.812761 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:36.813484 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.814301 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.815495 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:36.815894 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:36.816021 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:36.820235 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53989: BlockingRecv error: recv got EOF from 127.0.97.2:53989 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:37.254804 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:37.246680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:34329)
0619 01:00:37.247147 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:37.247185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:37.247343 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:37.247677 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:37.247681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:37.248061 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:37.248228 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.248237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.249216 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.249224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.250172 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.250184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.250949 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.250957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.251120 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.251681 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:37.251700 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:37.254476 (+ 2776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
W20260619 01:00:37.254923 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:37.246779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34329 (local address 127.0.97.62:36149)
0619 01:00:37.246967 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:37.246975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:37.247273 (+ 298us) server_negotiation.cc:407] Connection header received
0619 01:00:37.247469 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:37.247473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:37.247551 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:37.247665 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:37.248375 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.249082 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.250314 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.250779 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.250904 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.254672 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34329: BlockingRecv error: recv got EOF from 127.0.97.3:34329 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:37.374327 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:37.366118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:55625)
0619 01:00:37.366593 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:37.366609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:37.366758 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:37.367060 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:37.367063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:37.367395 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:37.367532 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.367540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.368516 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.368519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.369462 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.369472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.370098 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.370102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.370232 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.370786 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:37.370803 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:37.373909 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":247,"threads_started":1}
W20260619 01:00:37.374414 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:37.366183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55625 (local address 127.0.97.62:36149)
0619 01:00:37.366430 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:37.366437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:37.366688 (+ 251us) server_negotiation.cc:407] Connection header received
0619 01:00:37.366868 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:37.366871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:37.366936 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:37.367023 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:37.367700 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.368392 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.369589 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.369981 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.370074 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.374108 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55625: BlockingRecv error: recv got EOF from 127.0.97.1:55625 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:37 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:37.829078 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:37.821177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:52159)
0619 01:00:37.821664 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:37.821690 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:37.821798 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:37.822101 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:37.822104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:37.822442 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:37.822586 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.822594 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.823663 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.823666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.824575 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:37.824585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.825213 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.825217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:37.825332 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.825873 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:37.825889 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:37.828850 (+ 2961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
W20260619 01:00:37.829149 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:37.821251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52159 (local address 127.0.97.62:36149)
0619 01:00:37.821494 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:37.821501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:37.821775 (+ 274us) server_negotiation.cc:407] Connection header received
0619 01:00:37.821901 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:37.821905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:37.821972 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:37.822065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:37.822772 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.823535 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.824687 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:37.825101 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:37.825197 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:37.828967 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52159: BlockingRecv error: recv got EOF from 127.0.97.2:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:38.264241 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:38.255858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:50457)
0619 01:00:38.256413 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:38.256433 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:38.256588 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:38.256962 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:38.256966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:38.257363 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:38.257519 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.257527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.258545 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.258554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.259674 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.259696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.260390 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.260396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.260588 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.261243 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:38.261260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:38.263918 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":296,"threads_started":1}
W20260619 01:00:38.264355 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:38.255924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50457 (local address 127.0.97.62:36149)
0619 01:00:38.256137 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:38.256146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:38.256519 (+ 373us) server_negotiation.cc:407] Connection header received
0619 01:00:38.256729 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:38.256734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:38.256819 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:38.256932 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:38.257663 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.258421 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.259829 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.260250 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.260381 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.264105 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50457: BlockingRecv error: recv got EOF from 127.0.97.3:50457 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:38.383069 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:38.375233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:48771)
0619 01:00:38.375756 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:38.375773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:38.375875 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:38.376165 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:38.376169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:38.376532 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:38.376693 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.376701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.377658 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.377661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.378546 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.378556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.379223 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.379227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.379337 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.379860 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:38.379875 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:38.382737 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":256,"threads_started":1}
W20260619 01:00:38.383078 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:38.375351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48771 (local address 127.0.97.62:36149)
0619 01:00:38.375526 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:38.375530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:38.375851 (+ 321us) server_negotiation.cc:407] Connection header received
0619 01:00:38.375970 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:38.375974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:38.376042 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:38.376124 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:38.376806 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.377538 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.378666 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.379093 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.379225 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.382887 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48771: BlockingRecv error: recv got EOF from 127.0.97.1:48771 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:38 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:38.838495 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:38.830182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:36011)
0619 01:00:38.830697 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:38.830732 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:38.830860 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:38.831152 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:38.831156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:38.831645 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:38.831827 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.831836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.833083 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.833090 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.834015 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:38.834026 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.834698 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.834702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:38.834871 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.835397 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:38.835414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:38.838256 (+ 2842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":300,"threads_started":1}
W20260619 01:00:38.838655 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:38.830352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36011 (local address 127.0.97.62:36149)
0619 01:00:38.830522 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:38.830526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:38.830805 (+ 279us) server_negotiation.cc:407] Connection header received
0619 01:00:38.830955 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:38.830959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:38.831028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:38.831108 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:38.831965 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.832884 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.834131 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:38.834573 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:38.834660 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:38.838423 (+ 3763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36011: BlockingRecv error: recv got EOF from 127.0.97.2:36011 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:39.273404 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:39.265411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:57569)
0619 01:00:39.265997 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:39.266024 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:39.266195 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:39.266612 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:39.266617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:39.267183 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:39.267354 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.267363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.268297 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.268301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.269226 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.269238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.269906 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.269910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.270033 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.270606 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:39.270622 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:39.273171 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":279,"threads_started":1}
W20260619 01:00:39.273537 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:39.265467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57569 (local address 127.0.97.62:36149)
0619 01:00:39.265661 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:39.265667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:39.266109 (+ 442us) server_negotiation.cc:407] Connection header received
0619 01:00:39.266330 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:39.266333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:39.266441 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:39.266546 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:39.267481 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.268160 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.269351 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.269736 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.269834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.273329 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57569: BlockingRecv error: recv got EOF from 127.0.97.3:57569 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:39.391707 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:39.383971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:42699)
0619 01:00:39.384391 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:39.384407 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:39.384502 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:39.385020 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:39.385023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:39.385383 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:39.385525 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.385532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.386507 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.386511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.387474 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.387484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.388158 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.388162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.388277 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.388816 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:39.388832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:39.391490 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
W20260619 01:00:39.391862 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:39.384610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42699 (local address 127.0.97.62:36149)
0619 01:00:39.384779 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:39.384783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:39.384797 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:00:39.384836 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:39.384840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:39.384893 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:39.384971 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:39.385652 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.386398 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.387601 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.388031 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.388137 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.391649 (+ 3512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42699: BlockingRecv error: recv got EOF from 127.0.97.1:42699 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:39 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:39.847954 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:39.839427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:45055)
0619 01:00:39.839951 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:39.839968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:39.840077 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:39.840327 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:39.840330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:39.840748 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:39.840893 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.840901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.841998 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.842002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.842974 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:39.842985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.843893 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.843897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:39.844012 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.844619 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:39.844635 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:39.847657 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
W20260619 01:00:39.848259 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:39.839471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45055 (local address 127.0.97.62:36149)
0619 01:00:39.839629 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:39.839633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:39.840037 (+ 404us) server_negotiation.cc:407] Connection header received
0619 01:00:39.840170 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:39.840173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:39.840236 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:39.840314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:39.841010 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.841878 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.843108 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:39.843756 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:39.843868 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:39.847940 (+ 4072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:45055: BlockingRecv error: recv got EOF from 127.0.97.2:45055 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:40.281615 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:40.274248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:35927)
0619 01:00:40.274703 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:40.274775 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:40.274880 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:40.275151 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:40.275153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:40.275512 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:40.275668 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.275685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.276634 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.276638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.277500 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.277510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.278128 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.278132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.278245 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.278854 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:40.278870 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:40.281403 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":249,"threads_started":1}
W20260619 01:00:40.281814 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:40.274336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35927 (local address 127.0.97.62:36149)
0619 01:00:40.274562 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:40.274567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:40.274846 (+ 279us) server_negotiation.cc:407] Connection header received
0619 01:00:40.274991 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:40.274995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:40.275063 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:40.275139 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:40.275790 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.276519 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.277613 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.278005 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.278097 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.281576 (+ 3479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35927: BlockingRecv error: recv got EOF from 127.0.97.3:35927 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:40.401054 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:40.392645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:53263)
0619 01:00:40.393238 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:40.393254 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:40.393422 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:40.393777 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:40.393781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:40.394181 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:40.394343 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.394351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.395437 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.395443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.396369 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.396380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.397084 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.397090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.397272 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.397805 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:40.397822 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:40.400670 (+ 2848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":252,"threads_started":1}
W20260619 01:00:40.401077 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:40.392747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53263 (local address 127.0.97.62:36149)
0619 01:00:40.392947 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:40.392956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:40.393344 (+ 388us) server_negotiation.cc:407] Connection header received
0619 01:00:40.393548 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:40.393553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:40.393626 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:40.393746 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:40.394487 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.395296 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.396510 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.396928 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.397055 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.400858 (+ 3803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53263: BlockingRecv error: recv got EOF from 127.0.97.1:53263 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:40 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:40.856618 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:40.848927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:60639)
0619 01:00:40.849485 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:40.849502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:40.849611 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:40.849921 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:40.849924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:40.850455 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:40.850627 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.850636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.851599 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.851603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.852462 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:40.852473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.853098 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.853102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:40.853222 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.853766 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:40.853783 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:40.856386 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
W20260619 01:00:40.856863 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:40.849004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60639 (local address 127.0.97.62:36149)
0619 01:00:40.849279 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:40.849285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:40.849583 (+ 298us) server_negotiation.cc:407] Connection header received
0619 01:00:40.849712 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:40.849716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:40.849787 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:40.849876 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:40.850764 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.851486 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.852573 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:40.852975 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:40.853064 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:40.856617 (+ 3553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60639: BlockingRecv error: recv got EOF from 127.0.97.2:60639 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:41.290793 9625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:41.282651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:56001)
0619 01:00:41.283474 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:41.283494 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:41.283619 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:41.283923 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:41.283926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:41.284283 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:41.284446 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.284454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.285573 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.285577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.286479 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.286490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.287219 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.287223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.287327 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.287835 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:41.287851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:41.290515 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":450,"threads_started":1}
W20260619 01:00:41.290990 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:41.282844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56001 (local address 127.0.97.62:36149)
0619 01:00:41.283070 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:41.283079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:41.283574 (+ 495us) server_negotiation.cc:407] Connection header received
0619 01:00:41.283728 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:41.283732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:41.283801 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:41.283880 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:41.284559 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.285439 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.286598 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.287094 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.287193 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.290783 (+ 3590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56001: BlockingRecv error: recv got EOF from 127.0.97.3:56001 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:41.414570 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:41.402134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:54145)
0619 01:00:41.402605 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:41.402621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:41.402794 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:41.403086 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:41.403089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:41.403453 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:41.403608 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.403617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.404591 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.404595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.405473 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.405483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.406203 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.406207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.406317 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.411548 (+ 5231us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:41.411565 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:41.414350 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":277,"threads_started":1}
W20260619 01:00:41.414779 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:41.402336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54145 (local address 127.0.97.62:36149)
0619 01:00:41.402539 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:41.402545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:41.402696 (+ 151us) server_negotiation.cc:407] Connection header received
0619 01:00:41.402902 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:41.402906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:41.402966 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:41.403047 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:41.403746 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.404474 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.405611 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.406053 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.406155 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.414533 (+ 8378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54145: BlockingRecv error: recv got EOF from 127.0.97.1:54145 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:41 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:41.866112 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:41.857602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:52899)
0619 01:00:41.858043 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:41.858059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:41.858170 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:41.858467 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:41.858470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:41.858909 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:41.859049 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.859057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.860731 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.860735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.861646 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:41.861656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.862334 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.862338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:41.862453 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.863062 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:41.863079 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:41.865794 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":235,"threads_started":1}
W20260619 01:00:41.866163 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:41.857678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52899 (local address 127.0.97.62:36149)
0619 01:00:41.857905 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:41.857912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:41.858139 (+ 227us) server_negotiation.cc:407] Connection header received
0619 01:00:41.858285 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:41.858289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:41.858346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:41.858423 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:41.859228 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.860597 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.861766 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:41.862168 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:41.862264 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:41.865965 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52899: BlockingRecv error: recv got EOF from 127.0.97.2:52899 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:42.300271 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:42.291943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:46709)
0619 01:00:42.292418 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:42.292439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:42.292550 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:42.292901 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:42.292904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:42.293285 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:42.293429 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.293437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.294798 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.294802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.295791 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.295803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.296584 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.296590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.296707 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.297332 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:42.297348 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:42.300062 (+ 2714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":270,"threads_started":1}
W20260619 01:00:42.300465 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:42.292010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46709 (local address 127.0.97.62:36149)
0619 01:00:42.292238 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:42.292244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:42.292518 (+ 274us) server_negotiation.cc:407] Connection header received
0619 01:00:42.292652 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:42.292657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:42.292770 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:42.292869 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:42.293564 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.294631 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.295911 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.296433 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.296528 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.300230 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46709: BlockingRecv error: recv got EOF from 127.0.97.3:46709 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:42.423521 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:42.415464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:55861)
0619 01:00:42.416007 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:42.416023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:42.416129 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:42.416410 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:42.416413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:42.416772 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:42.416934 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.416942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.418104 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.418110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.419405 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.419417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.420032 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.420035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.420147 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.420664 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:42.420680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:42.423289 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":297,"threads_started":1}
W20260619 01:00:42.423676 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:42.415490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55861 (local address 127.0.97.62:36149)
0619 01:00:42.415633 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:42.415638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:42.416084 (+ 446us) server_negotiation.cc:407] Connection header received
0619 01:00:42.416219 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:42.416222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:42.416294 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:42.416372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:42.417040 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.417952 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.419522 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.419916 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.420009 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.423464 (+ 3455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55861: BlockingRecv error: recv got EOF from 127.0.97.1:55861 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:42 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:42.875413 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:42.867005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:33949)
0619 01:00:42.867535 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:42.867551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:42.867657 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:42.867893 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:42.867896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:42.868277 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:42.868453 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.868461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.869459 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.869463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.870419 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:42.870430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.871169 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.871174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:42.871344 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.872056 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:42.872081 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:42.875055 (+ 2974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":299,"threads_started":1}
W20260619 01:00:42.875451 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:42.867040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33949 (local address 127.0.97.62:36149)
0619 01:00:42.867204 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:42.867209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:42.867620 (+ 411us) server_negotiation.cc:407] Connection header received
0619 01:00:42.867745 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:42.867749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:42.867804 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:42.867881 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:42.868566 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.869350 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.870545 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:42.871024 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:42.871121 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:42.875224 (+ 4103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33949: BlockingRecv error: recv got EOF from 127.0.97.2:33949 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:43.309333 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:43.301201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:46599)
0619 01:00:43.301667 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:43.301689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:43.301846 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:43.302213 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:43.302217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:43.302645 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:43.302923 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.302934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.304093 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.304097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.305014 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.305024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.305606 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.305610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.305733 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.306304 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:43.306320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:43.308978 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
W20260619 01:00:43.309392 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:43.301242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46599 (local address 127.0.97.62:36149)
0619 01:00:43.301444 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:43.301451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:43.301771 (+ 320us) server_negotiation.cc:407] Connection header received
0619 01:00:43.301972 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:43.301977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:43.302073 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:43.302174 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:43.303055 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.303965 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.305120 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.305487 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.305573 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.309173 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46599: BlockingRecv error: recv got EOF from 127.0.97.3:46599 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:43.432739 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:43.424418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:41757)
0619 01:00:43.424917 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:43.424932 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:43.425078 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:43.425413 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:43.425417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:43.425792 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:43.425925 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.425933 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.427014 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.427022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.427958 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.427968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.428694 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.428699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.428855 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.429408 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:43.429424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:43.432507 (+ 3083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":277,"threads_started":1}
W20260619 01:00:43.432935 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:43.424485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41757 (local address 127.0.97.62:36149)
0619 01:00:43.424760 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:43.424767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:43.425017 (+ 250us) server_negotiation.cc:407] Connection header received
0619 01:00:43.425222 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:43.425227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:43.425292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:43.425388 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:43.426092 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.426889 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.428103 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.428534 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.428662 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.432724 (+ 4062us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41757: BlockingRecv error: recv got EOF from 127.0.97.1:41757 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:43 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:43.884053 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:43.876276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:53501)
0619 01:00:43.876747 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:43.876764 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:43.876906 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:43.877237 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:43.877245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:43.877609 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:43.877749 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.877758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.878753 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.878760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.879670 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:43.879680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.880341 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.880347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:43.880504 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.881039 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:43.881056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:43.883833 (+ 2777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":263,"threads_started":1}
W20260619 01:00:43.884275 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:43.876350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53501 (local address 127.0.97.62:36149)
0619 01:00:43.876580 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:43.876590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:43.876848 (+ 258us) server_negotiation.cc:407] Connection header received
0619 01:00:43.877021 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:43.877026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:43.877092 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:43.877188 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:43.877893 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.878600 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.879799 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:43.880203 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:43.880328 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:43.884013 (+ 3685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53501: BlockingRecv error: recv got EOF from 127.0.97.2:53501 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:44.318142 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:44.310141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:56647)
0619 01:00:44.310652 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:44.310672 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:44.310854 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:44.311187 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:44.311191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:44.311560 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:44.311696 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.311705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.312752 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.312761 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.313690 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.313703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.314407 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.314413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.314578 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.315130 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:44.315146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:44.317891 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
W20260619 01:00:44.318344 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:44.310307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56647 (local address 127.0.97.62:36149)
0619 01:00:44.310548 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:44.310554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:44.310765 (+ 211us) server_negotiation.cc:407] Connection header received
0619 01:00:44.310967 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:44.310973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:44.311064 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:44.311157 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:44.311830 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.312601 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.313833 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.314261 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.314385 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.318100 (+ 3715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56647: BlockingRecv error: recv got EOF from 127.0.97.3:56647 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:44.441478 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:44.433755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:43521)
0619 01:00:44.434317 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:44.434333 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:44.434435 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:44.434680 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:44.434689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:44.435095 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:44.435257 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.435266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.436210 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.436213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.437207 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.437218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.437982 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.437987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.438106 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.438645 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:44.438663 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:44.441252 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":340,"threads_started":1}
W20260619 01:00:44.441608 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:44.433777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43521 (local address 127.0.97.62:36149)
0619 01:00:44.433939 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:44.433943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:44.434403 (+ 460us) server_negotiation.cc:407] Connection header received
0619 01:00:44.434526 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:44.434530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:44.434597 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:44.434667 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:44.435368 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.436085 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.437324 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.437844 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.437936 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.441431 (+ 3495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43521: BlockingRecv error: recv got EOF from 127.0.97.1:43521 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:44 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:44.892951 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:44.884909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:58861)
0619 01:00:44.885359 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:44.885375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:44.885485 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:44.885747 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:44.885750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:44.886096 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:44.886284 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.886292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.887328 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.887332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.888622 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:44.888635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.889307 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.889311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:44.889414 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.889985 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:44.890001 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:44.892721 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
W20260619 01:00:44.893103 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:44.884971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58861 (local address 127.0.97.62:36149)
0619 01:00:44.885179 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:44.885185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:44.885451 (+ 266us) server_negotiation.cc:407] Connection header received
0619 01:00:44.885576 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:44.885580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:44.885639 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:44.885729 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:44.886401 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.887174 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.888740 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:44.889137 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:44.889255 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:44.892889 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58861: BlockingRecv error: recv got EOF from 127.0.97.2:58861 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:45.326572 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:45.319011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:35615)
0619 01:00:45.319483 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:45.319503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:45.319610 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:45.319874 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:45.319877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:45.320257 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:45.320426 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.320434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.321372 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.321375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.322260 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.322271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.322891 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.322896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.323013 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.323642 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:45.323659 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:45.326341 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
W20260619 01:00:45.326713 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:45.319056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35615 (local address 127.0.97.62:36149)
0619 01:00:45.319232 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:45.319237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:45.319571 (+ 334us) server_negotiation.cc:407] Connection header received
0619 01:00:45.319700 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:45.319703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:45.319761 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:45.319838 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:45.320540 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.321268 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.322371 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.322773 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.322866 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.326521 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35615: BlockingRecv error: recv got EOF from 127.0.97.3:35615 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:45.451344 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:45.442504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:33093)
0619 01:00:45.443056 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:45.443072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:45.443234 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:45.443577 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:45.443582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:45.444022 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:45.444180 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.444190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.445270 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.445278 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.446269 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.446280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.447148 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.447155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.447339 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.448139 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:45.448164 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:45.451100 (+ 2936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":321,"threads_started":1}
W20260619 01:00:45.451478 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:45.442543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33093 (local address 127.0.97.62:36149)
0619 01:00:45.442801 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:45.442809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:45.443163 (+ 354us) server_negotiation.cc:407] Connection header received
0619 01:00:45.443363 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:45.443368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:45.443437 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:45.443545 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:45.444334 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.445115 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.446414 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.447003 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.447131 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.451273 (+ 4142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:33093: BlockingRecv error: recv got EOF from 127.0.97.1:33093 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:45 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:45.901455 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:45.893776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:52679)
0619 01:00:45.894212 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:45.894228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:45.894334 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:45.894597 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:45.894600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:45.894987 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:45.895126 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.895134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.896066 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.896069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.896932 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:45.896941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.897545 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.897548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:45.897660 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.898187 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:45.898203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:45.901250 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
W20260619 01:00:45.901588 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:45.893846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52679 (local address 127.0.97.62:36149)
0619 01:00:45.894071 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:45.894077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:45.894297 (+ 220us) server_negotiation.cc:407] Connection header received
0619 01:00:45.894422 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:45.894426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:45.894483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:45.894558 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:45.895250 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.895954 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.897038 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:45.897439 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:45.897533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:45.901385 (+ 3852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52679: BlockingRecv error: recv got EOF from 127.0.97.2:52679 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:46.335472 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:46.327454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:51831)
0619 01:00:46.328022 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:46.328041 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:46.328145 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:46.328409 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:46.328412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:46.328755 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:46.328902 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.328910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.329967 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.329971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.330939 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.330951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.331641 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.331645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.331771 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.332393 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:46.332410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:46.335138 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
W20260619 01:00:46.335623 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:46.327491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51831 (local address 127.0.97.62:36149)
0619 01:00:46.327645 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:46.327650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:46.328106 (+ 456us) server_negotiation.cc:407] Connection header received
0619 01:00:46.328234 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:46.328238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:46.328292 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:46.328370 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:46.329047 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.329818 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.331085 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.331520 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.331620 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.335373 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51831: BlockingRecv error: recv got EOF from 127.0.97.3:51831 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:46.464874 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:46.452218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:45005)
0619 01:00:46.452734 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:46.452750 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:46.452891 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:46.453268 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:46.453272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:46.453799 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:46.454012 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.454026 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.455071 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.455075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.455974 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.455985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.456624 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.456628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.456741 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.461880 (+ 5139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:46.461897 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:46.464579 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":267,"threads_started":1}
W20260619 01:00:46.465049 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:46.452272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45005 (local address 127.0.97.62:36149)
0619 01:00:46.452442 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:46.452446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:46.452840 (+ 394us) server_negotiation.cc:407] Connection header received
0619 01:00:46.453022 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:46.453027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:46.453103 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:46.453201 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:46.454170 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.454965 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.456088 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.456490 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.456576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.464792 (+ 8216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45005: BlockingRecv error: recv got EOF from 127.0.97.1:45005 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:46 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:46.910765 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:46.902324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:57097)
0619 01:00:46.902857 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:46.902872 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:46.902978 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:46.903357 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:46.903360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:46.903749 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:46.903901 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.903909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.905043 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.905046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.905938 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:46.905948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.906679 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.906683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:46.906865 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.907474 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:46.907491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:46.910316 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":265,"threads_started":1}
W20260619 01:00:46.910796 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:46.902404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57097 (local address 127.0.97.62:36149)
0619 01:00:46.902558 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:46.902562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:46.902943 (+ 381us) server_negotiation.cc:407] Connection header received
0619 01:00:46.903159 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:46.903163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:46.903229 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:46.903324 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:46.904032 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.904919 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.906068 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:46.906535 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:46.906670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:46.910533 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57097: BlockingRecv error: recv got EOF from 127.0.97.2:57097 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:47.344264 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:47.336336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:49577)
0619 01:00:47.336855 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:47.336873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:47.337079 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:47.337417 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:47.337423 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:47.337909 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:47.338110 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.338124 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.339147 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.339152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.340060 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.340070 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.340797 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.340802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.340908 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.341449 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:47.341473 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:47.344041 (+ 2568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":293,"threads_started":1}
W20260619 01:00:47.344468 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:47.336503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49577 (local address 127.0.97.62:36149)
0619 01:00:47.336665 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:47.336672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:47.336957 (+ 285us) server_negotiation.cc:407] Connection header received
0619 01:00:47.337204 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:47.337208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:47.337267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:47.337366 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:47.338252 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.339009 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.340182 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.340643 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.340737 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.344212 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49577: BlockingRecv error: recv got EOF from 127.0.97.3:49577 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:47.473661 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:47.465827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:41365)
0619 01:00:47.466303 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:47.466319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:47.466466 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:47.466840 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:47.466844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:47.467244 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:47.467404 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.467412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.468403 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.468408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.469309 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.469320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.469991 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.469996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.470163 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.470807 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:47.470823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:47.473427 (+ 2604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":248,"threads_started":1}
W20260619 01:00:47.473826 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:47.465885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41365 (local address 127.0.97.62:36149)
0619 01:00:47.466191 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:47.466197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:47.466399 (+ 202us) server_negotiation.cc:407] Connection header received
0619 01:00:47.466591 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:47.466596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:47.466660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:47.466813 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:47.467535 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.468272 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.469446 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.469859 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.469980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.473595 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41365: BlockingRecv error: recv got EOF from 127.0.97.1:41365 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:47 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:47.920032 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:47.911739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:43331)
0619 01:00:47.912367 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:47.912384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:47.912498 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:47.912871 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:47.912877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:47.913341 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:47.913567 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.913580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.914778 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.914790 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.915721 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:47.915732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.916398 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.916403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:47.916574 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.917244 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:47.917261 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:47.919818 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":319,"threads_started":1}
W20260619 01:00:47.920218 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:47.911804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43331 (local address 127.0.97.62:36149)
0619 01:00:47.911983 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:47.911987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:47.912464 (+ 477us) server_negotiation.cc:407] Connection header received
0619 01:00:47.912609 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:47.912614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:47.912697 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:47.912821 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:47.913728 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.914628 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.915857 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:47.916268 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:47.916388 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:47.919987 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43331: BlockingRecv error: recv got EOF from 127.0.97.2:43331 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:48.353399 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:48.345285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.3:55285)
0619 01:00:48.345757 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:48.345777 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:48.345888 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:48.346161 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:48.346164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:48.346502 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:48.346665 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.346672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.347623 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.347627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.348534 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.348545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.349373 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.349379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.349545 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.350399 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:48.350423 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:48.353141 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":254,"threads_started":1}
W20260619 01:00:48.353567 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:48.345313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55285 (local address 127.0.97.62:36149)
0619 01:00:48.345478 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:48.345482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:48.345854 (+ 372us) server_negotiation.cc:407] Connection header received
0619 01:00:48.345988 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:48.345992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:48.346049 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:48.346129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:48.346815 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.347493 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.348651 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.349219 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.349345 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.353351 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55285: BlockingRecv error: recv got EOF from 127.0.97.3:55285 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:48.482832 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:48.474619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.1:48253)
0619 01:00:48.475137 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:48.475152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:48.475265 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:48.475532 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:48.475535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:48.475943 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:48.476189 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.476200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.477498 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.477502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.478390 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.478401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.479083 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.479087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.479194 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.479745 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:48.479760 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:48.482423 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":305,"threads_started":1}
W20260619 01:00:48.482851 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:48.474761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48253 (local address 127.0.97.62:36149)
0619 01:00:48.474933 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:48.474937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:48.475228 (+ 291us) server_negotiation.cc:407] Connection header received
0619 01:00:48.475358 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:48.475362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:48.475419 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:48.475498 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:48.476395 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.477358 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.478519 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.478976 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.479069 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.482602 (+ 3533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48253: BlockingRecv error: recv got EOF from 127.0.97.1:48253 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 01:00:48 dist-test-slave-3rxd krb5kdc[8603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 01:00:48.929159 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:48.921192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36149 (local address 127.0.97.2:54449)
0619 01:00:48.921639 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:48.921656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:48.921775 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:48.922066 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:48.922070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:48.922607 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:48.922895 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.922911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.924046 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.924049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.924972 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:48.924983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.925637 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.925641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:48.925755 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.926282 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:48.926298 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:48.928924 (+ 2626us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:36149: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":236,"threads_started":1}
W20260619 01:00:48.929281 9340 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:48.921251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54449 (local address 127.0.97.62:36149)
0619 01:00:48.921466 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:48.921472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:48.921743 (+ 271us) server_negotiation.cc:407] Connection header received
0619 01:00:48.921871 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:48.921875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:48.921934 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:48.922013 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:48.923036 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.923921 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.925083 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:48.925514 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:48.925601 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:48.929081 (+ 3480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54449: BlockingRecv error: recv got EOF from 127.0.97.2:54449 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:00:49.201323 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9190
I20260619 01:00:49.212621 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:49.202344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40672 (local address 127.0.97.1:34209)
0619 01:00:49.202919 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:49.202926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:49.202950 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:00:49.203005 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:49.203011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:49.203249 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:49.203396 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:49.204668 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.205659 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.206785 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.207303 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.207394 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:49.209643 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:49.209651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:49.209663 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:49.209694 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:49.212013 (+ 2319us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:00:49.212325 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40672: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
W20260619 01:00:49.212760 9650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40672: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 01:00:49.212730 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9190: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:34209: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:00:49.212932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9190
I20260619 01:00:49.228960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9341
W20260619 01:00:49.240422 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9341: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:41841: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:00:49.240491 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:49.229880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.0.97.2:41841)
0619 01:00:49.230422 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:49.230429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:49.230457 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:00:49.230510 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:49.230516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:49.230777 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:49.230958 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:49.232182 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.233249 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.234424 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.235040 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.235179 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:49.237377 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:49.237386 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:49.237398 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:49.237430 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:49.239766 (+ 2336us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:00:49.240113 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38314: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":263,"threads_started":1}
I20260619 01:00:49.240576 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9341
W20260619 01:00:49.240832 9651 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38314: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:00:49.257884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9472
I20260619 01:00:49.268958 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:49.258688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.0.97.3:45891)
0619 01:00:49.259319 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:49.259326 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:49.259351 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:00:49.259407 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:49.259413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:49.259620 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:49.259772 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:49.260857 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.261872 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.263118 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.263624 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.263726 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:49.265951 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:49.265958 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:49.265971 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:49.266009 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:49.268321 (+ 2312us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 01:00:49.268619 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58544: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":298,"threads_started":1}
W20260619 01:00:49.268980 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9472: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45891: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 01:00:49.269063 9652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58544: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 01:00:49.269119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9472
I20260619 01:00:49.284750 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9121
I20260619 01:00:49.285856 9183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:49.398300 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9121
2026-06-19T01:00:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20860 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T01:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:49Z Disabled control of system clock
I20260619 01:00:49.470243 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35489
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 01:00:49.790421 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:49.790701 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:49.790849 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:49.797785 9657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:49.797899 9657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:49.797986 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:49.798080 9657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:49.798166 9657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:49.806839 9657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:49.808048 9657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:49.809773 9657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:49.821470 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:49.821511 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:49.821547 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:49.821940 9657 server_base.cc:1034] running on GCE node
I20260619 01:00:49.822661 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:49.823982 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:49.825196 9657 hybrid_clock.cc:648] HybridClock initialized: now 1781830849825156 us; error 49 us; skew 500 ppm
I20260619 01:00:49.825528 9657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:49.827689 9657 webserver.cc:460] Webserver started at http://127.0.97.62:40307/ using document root <none> and password file <none>
I20260619 01:00:49.828346 9657 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:49.828469 9657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:49.828800 9657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:49.831532 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "575a06c52c1f4d0d93f2ecb4754600c0"
format_stamp: "Formatted at 2026-06-19 01:00:49 on dist-test-slave-3rxd"
server_key: "1ae42334d88b4b9ad617f7f50cf6fdaf"
server_key_iv: "114f402d4d5ce3791a0e6bca969d03d0"
server_key_version: "encryptionkey@0"
I20260619 01:00:49.832285 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "575a06c52c1f4d0d93f2ecb4754600c0"
format_stamp: "Formatted at 2026-06-19 01:00:49 on dist-test-slave-3rxd"
server_key: "1ae42334d88b4b9ad617f7f50cf6fdaf"
server_key_iv: "114f402d4d5ce3791a0e6bca969d03d0"
server_key_version: "encryptionkey@0"
I20260619 01:00:49.836989 9657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:00:49.840121 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:49.841318 9657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:00:49.841485 9657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "575a06c52c1f4d0d93f2ecb4754600c0"
format_stamp: "Formatted at 2026-06-19 01:00:49 on dist-test-slave-3rxd"
server_key: "1ae42334d88b4b9ad617f7f50cf6fdaf"
server_key_iv: "114f402d4d5ce3791a0e6bca969d03d0"
server_key_version: "encryptionkey@0"
I20260619 01:00:49.841683 9657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:49.865310 9657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:49.866112 9657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:49.866395 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:49.884238 9657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35489
I20260619 01:00:49.884258 9722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35489 every 8 connection(s)
I20260619 01:00:49.885496 9657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:49.887274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9657
I20260619 01:00:49.887449 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:49.888069 388 external_mini_cluster.cc:1428] Setting key 30ce091ef2a161b0fc3ddddf26dcd785
I20260619 01:00:49.889330 9723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:49.900501 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:49.889959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44678 (local address 127.0.97.62:35489)
0619 01:00:49.890950 (+ 991us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:49.890961 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:49.891001 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:00:49.892424 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:49.892444 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:49.892880 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:49.893261 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:49.893802 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.895145 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.896443 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:49.897063 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:49.897249 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:49.898374 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:49.898382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:49.898398 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:49.898968 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:49.899038 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:49.899057 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:49.899171 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:49.899388 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 01:00:49.899643 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":339,"threads_started":1}
I20260619 01:00:49.900584 9723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0: Bootstrap starting.
I20260619 01:00:49.903841 9723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:49.904714 9723 log.cc:824] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:49.907045 9723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0: No bootstrap required, opened a new log
I20260619 01:00:49.913806 9723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260619 01:00:49.914175 9723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:49.914268 9723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a06c52c1f4d0d93f2ecb4754600c0, State: Initialized, Role: FOLLOWER
I20260619 01:00:49.914875 9723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260619 01:00:49.915084 9723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:49.915213 9723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:49.915375 9723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:49.917266 9723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260619 01:00:49.917752 9723 leader_election.cc:302] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 575a06c52c1f4d0d93f2ecb4754600c0; no voters:
I20260619 01:00:49.918257 9723 leader_election.cc:288] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:49.918402 9728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:49.919086 9728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [term 1 LEADER]: Becoming Leader. State: Replica: 575a06c52c1f4d0d93f2ecb4754600c0, State: Running, Role: LEADER
I20260619 01:00:49.919579 9728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260619 01:00:49.920166 9723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:49.922369 9730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 575a06c52c1f4d0d93f2ecb4754600c0. Latest consensus state: current_term: 1 leader_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } } }
I20260619 01:00:49.922696 9730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:49.923483 9734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:49.923834 9729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575a06c52c1f4d0d93f2ecb4754600c0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } } }
I20260619 01:00:49.924225 9729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:49.928287 9734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:49.936661 9734 catalog_manager.cc:1347] Generated new cluster ID: 14568677b9aa4e2587651429a2955811
I20260619 01:00:49.936755 9734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:49.958535 9734 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:49.959694 9734 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:49.964757 9734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 575a06c52c1f4d0d93f2ecb4754600c0: Generated new TSK 0
I20260619 01:00:49.965441 9734 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:49.972209 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--builtin_ntp_servers=127.0.97.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 01:00:50.287688 9747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:50.287941 9747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:50.288020 9747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:50.297894 9747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:50.297983 9747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:50.298098 9747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:50.306646 9747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:50.307744 9747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:50.309345 9747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:50.321384 9755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:50.321388 9753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:50.321631 9752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:50.324186 9747 server_base.cc:1034] running on GCE node
I20260619 01:00:50.324784 9747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:50.325675 9747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:50.326882 9747 hybrid_clock.cc:648] HybridClock initialized: now 1781830850326837 us; error 42 us; skew 500 ppm
I20260619 01:00:50.327189 9747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:50.329463 9747 webserver.cc:460] Webserver started at http://127.0.97.1:41945/ using document root <none> and password file <none>
I20260619 01:00:50.330119 9747 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:50.330242 9747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:50.330544 9747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:50.333266 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "45b30148ffe148bb88970a548374be38"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "311abfefc9e8a49c8501190585fe9a66"
server_key_iv: "f05316ab005671fe7261b66f919daac1"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.334034 9747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "45b30148ffe148bb88970a548374be38"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "311abfefc9e8a49c8501190585fe9a66"
server_key_iv: "f05316ab005671fe7261b66f919daac1"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.338675 9747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 01:00:50.341810 9761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.343075 9747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:00:50.343227 9747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "45b30148ffe148bb88970a548374be38"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "311abfefc9e8a49c8501190585fe9a66"
server_key_iv: "f05316ab005671fe7261b66f919daac1"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.343426 9747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:50.361234 9747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:50.361925 9747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:50.362195 9747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:50.363066 9747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:50.364351 9747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:50.364472 9747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.364590 9747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:50.364674 9747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.395334 9747 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42617
I20260619 01:00:50.395408 9873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42617 every 8 connection(s)
I20260619 01:00:50.396533 9747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:50.397694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9747
I20260619 01:00:50.397882 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:50.398751 388 external_mini_cluster.cc:1428] Setting key 1b3095c5e3c28eb6af2b332fafd4b04c
I20260619 01:00:50.401880 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--builtin_ntp_servers=127.0.97.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 01:00:50.411164 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:50.399204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52225 (local address 127.0.97.62:35489)
0619 01:00:50.399406 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:50.399410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:50.401094 (+ 1684us) server_negotiation.cc:407] Connection header received
0619 01:00:50.403784 (+ 2690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:50.403787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:50.403846 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:50.403927 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:50.405171 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.405973 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.408436 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.408816 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.408906 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:50.410278 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:50.410287 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:50.410294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:50.410318 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:50.410358 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:50.410376 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:50.410461 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:50.410647 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 01:00:50.410780 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 01:00:50.411963 9876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:50.399945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35489 (local address 127.0.97.1:52225)
0619 01:00:50.400962 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:50.401007 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:50.403516 (+ 2509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:50.404136 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:50.404149 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:50.404261 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:50.405019 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:50.405042 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.406123 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.406131 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:50.408275 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:50.408303 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.409022 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.409031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:50.409286 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:50.409984 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:50.409987 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:50.410022 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:50.410096 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:50.410472 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:50.410479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:50.410484 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:50.410611 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 01:00:50.410935 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":294,"threads_started":1}
I20260619 01:00:50.413393 9874 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35489
I20260619 01:00:50.413667 9874 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:50.414361 9874 heartbeater.cc:511] Master 127.0.97.62:35489 requested a full tablet report, sending...
I20260619 01:00:50.416366 9688 ts_manager.cc:194] Registered new tserver with Master: 45b30148ffe148bb88970a548374be38 (127.0.97.1:42617)
I20260619 01:00:50.417807 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:52225
W20260619 01:00:50.712867 9877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:50.713153 9877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:50.713279 9877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:50.720376 9877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:50.720459 9877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:50.720624 9877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:50.729475 9877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:50.730640 9877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:50.732434 9877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:50.744890 9882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:50.744922 9885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:50.745489 9883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:50.745720 9877 server_base.cc:1034] running on GCE node
I20260619 01:00:50.746286 9877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:50.747256 9877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:50.748464 9877 hybrid_clock.cc:648] HybridClock initialized: now 1781830850748423 us; error 37 us; skew 500 ppm
I20260619 01:00:50.748754 9877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:50.750903 9877 webserver.cc:460] Webserver started at http://127.0.97.2:34991/ using document root <none> and password file <none>
I20260619 01:00:50.751541 9877 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:50.751663 9877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:50.751964 9877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:50.754577 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "dbdd864b48bd44ecab4b3ebb4e69b132"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "bb33f1e38c6fe93e2223ec32dd4c2192"
server_key_iv: "1f11a0821a278de88a63207ef3c607c0"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.755348 9877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "dbdd864b48bd44ecab4b3ebb4e69b132"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "bb33f1e38c6fe93e2223ec32dd4c2192"
server_key_iv: "1f11a0821a278de88a63207ef3c607c0"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.759945 9877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 01:00:50.763191 9891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.764441 9877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:50.764603 9877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "dbdd864b48bd44ecab4b3ebb4e69b132"
format_stamp: "Formatted at 2026-06-19 01:00:50 on dist-test-slave-3rxd"
server_key: "bb33f1e38c6fe93e2223ec32dd4c2192"
server_key_iv: "1f11a0821a278de88a63207ef3c607c0"
server_key_version: "encryptionkey@0"
I20260619 01:00:50.764799 9877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:50.781381 9877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:50.782208 9877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:50.782480 9877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:50.783457 9877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:50.784860 9877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:50.784977 9877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.785104 9877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:50.785192 9877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:50.816268 9877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36391
I20260619 01:00:50.816322 10003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36391 every 8 connection(s)
I20260619 01:00:50.817546 9877 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:50.824265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 9877
I20260619 01:00:50.824416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:50.824960 388 external_mini_cluster.cc:1428] Setting key 9119dbc9a645c3140809c618f7660bb8
I20260619 01:00:50.827560 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--builtin_ntp_servers=127.0.97.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 01:00:50.830883 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:50.820026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52033 (local address 127.0.97.62:35489)
0619 01:00:50.820192 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:50.820196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:50.821692 (+ 1496us) server_negotiation.cc:407] Connection header received
0619 01:00:50.823474 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:50.823479 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:50.823573 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:50.823678 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:50.825045 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.825830 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.827711 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.828473 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.828622 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:50.829985 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:50.829994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:50.830001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:50.830024 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:50.830063 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:50.830082 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:50.830174 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:50.830370 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 01:00:50.830474 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:50.831709 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:50.820596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35489 (local address 127.0.97.2:52033)
0619 01:00:50.821550 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:50.821591 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:50.823221 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:50.823941 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:50.823954 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:50.824072 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:50.824860 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:50.824883 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.825987 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.825992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:50.827545 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:50.827560 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:50.828685 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:50.828694 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:50.828997 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:50.829686 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:50.829688 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:50.829728 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:50.829794 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:50.830171 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:50.830178 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:50.830183 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:00:50.830333 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 01:00:50.830599 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":276,"threads_started":1}
I20260619 01:00:50.832998 10004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35489
I20260619 01:00:50.833285 10004 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:50.833981 10004 heartbeater.cc:511] Master 127.0.97.62:35489 requested a full tablet report, sending...
I20260619 01:00:50.835299 9688 ts_manager.cc:194] Registered new tserver with Master: dbdd864b48bd44ecab4b3ebb4e69b132 (127.0.97.2:36391)
I20260619 01:00:50.836058 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:52033
W20260619 01:00:51.138078 10007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:51.138432 10007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:51.138585 10007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:51.145578 10007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:51.145690 10007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:51.145843 10007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:51.154663 10007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:51.155884 10007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:51.157629 10007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:51.170286 10012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:51.170324 10013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:51.170306 10015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:51.171576 10007 server_base.cc:1034] running on GCE node
I20260619 01:00:51.172194 10007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:51.173131 10007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:51.174372 10007 hybrid_clock.cc:648] HybridClock initialized: now 1781830851174309 us; error 56 us; skew 500 ppm
I20260619 01:00:51.174683 10007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:51.176968 10007 webserver.cc:460] Webserver started at http://127.0.97.3:39619/ using document root <none> and password file <none>
I20260619 01:00:51.177925 10007 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:51.178056 10007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:51.178390 10007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:51.181269 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "2c291d5a289742c896067460ccfceee4"
format_stamp: "Formatted at 2026-06-19 01:00:51 on dist-test-slave-3rxd"
server_key: "89b442a2a1fa0b4e4dab36646526fa51"
server_key_iv: "20b0d69294b14b522b9cbf6f81a3ce98"
server_key_version: "encryptionkey@0"
I20260619 01:00:51.182040 10007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "2c291d5a289742c896067460ccfceee4"
format_stamp: "Formatted at 2026-06-19 01:00:51 on dist-test-slave-3rxd"
server_key: "89b442a2a1fa0b4e4dab36646526fa51"
server_key_iv: "20b0d69294b14b522b9cbf6f81a3ce98"
server_key_version: "encryptionkey@0"
I20260619 01:00:51.187422 10007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:00:51.190780 10021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:51.192010 10007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:00:51.192185 10007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "2c291d5a289742c896067460ccfceee4"
format_stamp: "Formatted at 2026-06-19 01:00:51 on dist-test-slave-3rxd"
server_key: "89b442a2a1fa0b4e4dab36646526fa51"
server_key_iv: "20b0d69294b14b522b9cbf6f81a3ce98"
server_key_version: "encryptionkey@0"
I20260619 01:00:51.192410 10007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:51.206964 10007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:51.207757 10007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:51.208019 10007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:51.208940 10007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:51.210371 10007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:51.210486 10007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:51.210609 10007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:51.210691 10007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:51.242403 10007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42787
I20260619 01:00:51.242453 10133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42787 every 8 connection(s)
I20260619 01:00:51.243697 10007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:51.249169 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10007
I20260619 01:00:51.249326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:51.249955 388 external_mini_cluster.cc:1428] Setting key a39e68888bd0216467811c4e4f0cd07b
I20260619 01:00:51.256839 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:51.246229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59171 (local address 127.0.97.62:35489)
0619 01:00:51.246380 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:51.246385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:51.247962 (+ 1577us) server_negotiation.cc:407] Connection header received
0619 01:00:51.250379 (+ 2417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:51.250385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:51.250477 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:51.250586 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:51.251907 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.252736 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.254274 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.254661 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.254832 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:51.256052 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:51.256058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:51.256063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:51.256086 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:51.256113 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:51.256124 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:51.256209 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:51.256438 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 01:00:51.256525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 01:00:51.257799 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:51.246834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35489 (local address 127.0.97.3:59171)
0619 01:00:51.247798 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:51.247844 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:51.250105 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:51.250887 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:51.250899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:51.251018 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:51.251752 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:51.251777 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.252876 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.252882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:51.254124 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:51.254136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.254915 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.254922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:51.255148 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:51.255765 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:51.255767 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:51.255804 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:51.255873 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:51.256235 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:51.256244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:51.256255 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:00:51.256414 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 01:00:51.256784 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":271,"threads_started":1}
I20260619 01:00:51.259076 10134 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35489
I20260619 01:00:51.259330 10134 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:51.259984 10134 heartbeater.cc:511] Master 127.0.97.62:35489 requested a full tablet report, sending...
I20260619 01:00:51.261233 9688 ts_manager.cc:194] Registered new tserver with Master: 2c291d5a289742c896067460ccfceee4 (127.0.97.3:42787)
I20260619 01:00:51.261943 9688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:59171
I20260619 01:00:51.264827 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 01:00:51.270594 9726 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:51.269065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.97.62:35489)
0619 01:00:51.269221 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:51.269225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:51.269680 (+ 455us) server_negotiation.cc:407] Connection header received
0619 01:00:51.269847 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:51.269851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:51.269935 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:51.270013 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:51.270388 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45686: BlockingRecv error: recv got EOF from 127.0.0.1:45686 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 01:00:51.271279 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9747
I20260619 01:00:51.278889 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:51.272265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.0.97.1:42617)
0619 01:00:51.272744 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:51.272751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:51.272773 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:00:51.272822 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:51.272827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:51.273014 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:51.273163 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:51.273547 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.274491 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.275739 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.276279 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.276402 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:51.277456 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:51.277465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:51.277476 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:51.278047 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:51.278091 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:51.278110 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:51.278236 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:51.278395 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 01:00:51.278521 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260619 01:00:51.280055 9869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:51.360651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9747
I20260619 01:00:51.377172 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9877
I20260619 01:00:51.384348 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:51.377978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.0.97.2:36391)
0619 01:00:51.378532 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:51.378539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:51.378566 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:00:51.378641 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:51.378647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:51.378934 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:51.379065 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:51.379412 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.380378 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.381506 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.382039 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.382130 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:51.383078 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:51.383086 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:51.383099 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:51.383683 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:51.383735 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:51.383754 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:51.383839 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:51.383950 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 01:00:51.384070 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":296,"threads_started":1}
I20260619 01:00:51.385547 9999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:51.464368 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9877
I20260619 01:00:51.480806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10007
I20260619 01:00:51.491541 10147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:51.481744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.0.97.3:42787)
0619 01:00:51.482348 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:51.482355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:51.482383 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:00:51.482443 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:51.482449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:51.482654 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:51.482887 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:51.483201 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.484207 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.485378 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:51.485894 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:51.485981 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:51.489904 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:51.489913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:51.489926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:51.490636 (+ 710us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:51.490744 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:51.490773 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:51.490892 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:51.491007 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 01:00:51.491169 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
I20260619 01:00:51.492766 10129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:51.576215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10007
I20260619 01:00:51.591753 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 9657
I20260619 01:00:51.592882 9718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:51.698297 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 9657
2026-06-19T01:00:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2295 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T01:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:51Z Disabled control of system clock
I20260619 01:00:51.768957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41711
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 01:00:52.089588 10153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:52.089895 10153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:52.089983 10153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:52.097355 10153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:52.097478 10153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:52.097523 10153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:52.097597 10153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:52.097649 10153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:52.107738 10153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41711
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:52.109076 10153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:52.110926 10153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:52.122462 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:52.122501 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:52.122434 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:52.123263 10153 server_base.cc:1034] running on GCE node
I20260619 01:00:52.124002 10153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:52.125406 10153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:52.126616 10153 hybrid_clock.cc:648] HybridClock initialized: now 1781830852126571 us; error 30 us; skew 500 ppm
I20260619 01:00:52.126978 10153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:52.129180 10153 webserver.cc:460] Webserver started at http://127.0.97.62:40277/ using document root <none> and password file <none>
I20260619 01:00:52.129829 10153 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:52.129930 10153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:52.130185 10153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:52.133060 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "1fbafec7ccf244b198794bcb296db184"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "157b0a2e63a6fa7633d59aa071e8e759"
server_key_iv: "99276ef1e16af78792bf069066053052"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.133795 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "1fbafec7ccf244b198794bcb296db184"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "157b0a2e63a6fa7633d59aa071e8e759"
server_key_iv: "99276ef1e16af78792bf069066053052"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.138653 10153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260619 01:00:52.141920 10167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:52.143235 10153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:00:52.143404 10153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "1fbafec7ccf244b198794bcb296db184"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "157b0a2e63a6fa7633d59aa071e8e759"
server_key_iv: "99276ef1e16af78792bf069066053052"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.143625 10153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:52.149011 10153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:52.149749 10153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:52.150050 10153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:52.168321 10153 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41711
I20260619 01:00:52.168350 10218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41711 every 8 connection(s)
I20260619 01:00:52.169704 10153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:52.173424 10219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:52.179769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10153
I20260619 01:00:52.179979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:52.180562 388 external_mini_cluster.cc:1428] Setting key 3f512004498cd05c19ffb08a5bc2cd73
I20260619 01:00:52.184167 10219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184: Bootstrap starting.
I20260619 01:00:52.188201 10219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:52.189529 10219 log.cc:824] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:52.190225 10222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:52.182514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.97.62:41711)
0619 01:00:52.183749 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:52.183764 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:52.183818 (+ 54us) server_negotiation.cc:407] Connection header received
0619 01:00:52.186151 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:52.186180 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:52.186760 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:52.187182 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:52.188179 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:52.188192 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:52.188209 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:52.188228 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:52.188304 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:52.188326 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:52.188530 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:52.188697 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 01:00:52.188950 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":385,"threads_started":1}
I20260619 01:00:52.192374 10219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184: No bootstrap required, opened a new log
I20260619 01:00:52.199290 10219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } }
I20260619 01:00:52.199635 10219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:52.199755 10219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbafec7ccf244b198794bcb296db184, State: Initialized, Role: FOLLOWER
I20260619 01:00:52.200361 10219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } }
I20260619 01:00:52.200587 10219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:52.200754 10219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:52.200935 10219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:52.203091 10219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } }
I20260619 01:00:52.203562 10219 leader_election.cc:302] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbafec7ccf244b198794bcb296db184; no voters:
I20260619 01:00:52.204082 10219 leader_election.cc:288] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:52.204192 10224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:52.205365 10224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [term 1 LEADER]: Becoming Leader. State: Replica: 1fbafec7ccf244b198794bcb296db184, State: Running, Role: LEADER
I20260619 01:00:52.205899 10224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } }
I20260619 01:00:52.206099 10219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:52.208899 10225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbafec7ccf244b198794bcb296db184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } } }
I20260619 01:00:52.209229 10225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:52.209566 10226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbafec7ccf244b198794bcb296db184. Latest consensus state: current_term: 1 leader_uuid: "1fbafec7ccf244b198794bcb296db184" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbafec7ccf244b198794bcb296db184" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41711 } } }
I20260619 01:00:52.209877 10226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184 [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:52.210832 10230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:52.215649 10230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:52.224498 10230 catalog_manager.cc:1347] Generated new cluster ID: cff47c156b864a3cb8efec7d3012abd2
I20260619 01:00:52.224620 10230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:52.242820 10230 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:52.243294 10230 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:52.258678 10230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fbafec7ccf244b198794bcb296db184: Generated new TSK 0
I20260619 01:00:52.259632 10230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:00:52.265344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--builtin_ntp_servers=127.0.97.20:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 01:00:52.615932 10243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:52.616199 10243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:52.616285 10243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:52.623382 10243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:52.623478 10243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:52.623584 10243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:52.632705 10243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:52.633875 10243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:52.635637 10243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:52.648566 10249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:52.648612 10248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:52.648630 10251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:52.649741 10243 server_base.cc:1034] running on GCE node
I20260619 01:00:52.650339 10243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:52.651306 10243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:52.652498 10243 hybrid_clock.cc:648] HybridClock initialized: now 1781830852652436 us; error 51 us; skew 500 ppm
I20260619 01:00:52.652810 10243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:52.655079 10243 webserver.cc:460] Webserver started at http://127.0.97.1:36379/ using document root <none> and password file <none>
I20260619 01:00:52.655745 10243 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:52.655853 10243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:52.656164 10243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:52.659401 10243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "54d109130f2c43dabeecc74f374d8b63"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "87119acbaf899791c5841bd551ae7950"
server_key_iv: "06a21ab7aeaa62f0a9778728d375d101"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.660151 10243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "54d109130f2c43dabeecc74f374d8b63"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "87119acbaf899791c5841bd551ae7950"
server_key_iv: "06a21ab7aeaa62f0a9778728d375d101"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.665130 10243 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:52.668491 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:52.669762 10243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:00:52.669902 10243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "54d109130f2c43dabeecc74f374d8b63"
format_stamp: "Formatted at 2026-06-19 01:00:52 on dist-test-slave-3rxd"
server_key: "87119acbaf899791c5841bd551ae7950"
server_key_iv: "06a21ab7aeaa62f0a9778728d375d101"
server_key_version: "encryptionkey@0"
I20260619 01:00:52.670089 10243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:52.675509 10243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:52.676196 10243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:52.676409 10243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:52.677205 10243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:52.678431 10243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:52.678514 10243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:52.678591 10243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:52.678645 10243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:52.710600 10243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43247
I20260619 01:00:52.710691 10369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43247 every 8 connection(s)
I20260619 01:00:52.711990 10243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:52.719659 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10243
I20260619 01:00:52.719873 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:52.720047 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:52.714331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59743 (local address 127.0.97.62:41711)
0619 01:00:52.714705 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:52.714764 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:52.716118 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 01:00:52.717809 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:52.717814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:52.717884 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:52.718022 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:52.719301 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:52.719307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:52.719311 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:52.719331 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:52.719374 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:52.719386 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:52.719505 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:52.719696 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 01:00:52.719770 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":189,"threads_started":1}
I20260619 01:00:52.720585 388 external_mini_cluster.cc:1428] Setting key ad3bb0e185a3bdbbefae31ff7b84537a
I20260619 01:00:52.720957 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:52.715048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41711 (local address 127.0.97.1:59743)
0619 01:00:52.715962 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:52.715998 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:52.717550 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:52.718258 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:52.718266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:52.718355 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:52.718986 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:52.718988 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:52.719023 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:52.719101 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:52.719488 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:52.719497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:52.719504 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 01:00:52.719778 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 01:00:52.720013 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
I20260619 01:00:52.722451 10370 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41711
I20260619 01:00:52.722793 10370 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:52.723237 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--builtin_ntp_servers=127.0.97.20:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 01:00:52.723361 10370 heartbeater.cc:511] Master 127.0.97.62:41711 requested a full tablet report, sending...
I20260619 01:00:52.725275 10184 ts_manager.cc:194] Registered new tserver with Master: 54d109130f2c43dabeecc74f374d8b63 (127.0.97.1:43247)
W20260619 01:00:53.043443 10374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:53.043701 10374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:53.043836 10374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:53.051112 10374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:53.051211 10374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:53.051379 10374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:53.060267 10374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:53.061511 10374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:53.063318 10374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:53.075585 10379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:53.075696 10380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:53.075726 10382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:53.077003 10374 server_base.cc:1034] running on GCE node
I20260619 01:00:53.077605 10374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:53.078508 10374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:53.079725 10374 hybrid_clock.cc:648] HybridClock initialized: now 1781830853079656 us; error 83 us; skew 500 ppm
I20260619 01:00:53.080018 10374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:53.082113 10374 webserver.cc:460] Webserver started at http://127.0.97.2:34417/ using document root <none> and password file <none>
I20260619 01:00:53.082830 10374 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:53.082952 10374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:53.083261 10374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:53.087770 10374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "fbd142410bd743b9938071cdee77e4b0"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "3e03b51c59677362ceac3d42d98d063c"
server_key_iv: "93afdd099e8558822380ca47b0f84981"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.088521 10374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fbd142410bd743b9938071cdee77e4b0"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "3e03b51c59677362ceac3d42d98d063c"
server_key_iv: "93afdd099e8558822380ca47b0f84981"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.093613 10374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:00:53.096917 10388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.098222 10374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 01:00:53.098420 10374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "fbd142410bd743b9938071cdee77e4b0"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "3e03b51c59677362ceac3d42d98d063c"
server_key_iv: "93afdd099e8558822380ca47b0f84981"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.098644 10374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:53.103982 10374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:53.104688 10374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:53.104941 10374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:53.105818 10374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:53.107359 10374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:53.107463 10374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.107585 10374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:53.107683 10374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.139391 10374 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37683
I20260619 01:00:53.139472 10500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37683 every 8 connection(s)
I20260619 01:00:53.140643 10374 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:53.143194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10374
I20260619 01:00:53.143383 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:53.143811 388 external_mini_cluster.cc:1428] Setting key 14299f36734d5948e4861768f3a72c16
I20260619 01:00:53.146371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--builtin_ntp_servers=127.0.97.20:34967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 01:00:53.149029 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.143254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35211 (local address 127.0.97.62:41711)
0619 01:00:53.143452 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.143458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.145020 (+ 1562us) server_negotiation.cc:407] Connection header received
0619 01:00:53.146631 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.146635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.146680 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.146795 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.148310 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:53.148316 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:53.148321 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:53.148338 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:53.148375 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:53.148386 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:53.148508 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:53.148702 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 01:00:53.148774 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:00:53.149911 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.143772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41711 (local address 127.0.97.2:35211)
0619 01:00:53.144857 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:53.144904 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:53.146341 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:53.147173 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:53.147180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:53.147277 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:53.148023 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:53.148025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:53.148062 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:53.148140 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:53.148476 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:53.148488 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:53.148497 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 01:00:53.148767 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 01:00:53.149006 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":404,"threads_started":1}
I20260619 01:00:53.151237 10501 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41711
I20260619 01:00:53.151533 10501 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:53.152186 10501 heartbeater.cc:511] Master 127.0.97.62:41711 requested a full tablet report, sending...
I20260619 01:00:53.153491 10184 ts_manager.cc:194] Registered new tserver with Master: fbd142410bd743b9938071cdee77e4b0 (127.0.97.2:37683)
W20260619 01:00:53.462745 10504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:53.463081 10504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:53.463248 10504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:53.470400 10504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:53.470563 10504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:53.470777 10504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:53.480180 10504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41711
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:53.481515 10504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:53.483326 10504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:53.495985 10509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:53.496023 10510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:53.496047 10512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:53.496968 10504 server_base.cc:1034] running on GCE node
I20260619 01:00:53.497565 10504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:53.498533 10504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:53.499754 10504 hybrid_clock.cc:648] HybridClock initialized: now 1781830853499711 us; error 53 us; skew 500 ppm
I20260619 01:00:53.500061 10504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:00:53.502254 10504 webserver.cc:460] Webserver started at http://127.0.97.3:44965/ using document root <none> and password file <none>
I20260619 01:00:53.503033 10504 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:53.503158 10504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:53.503474 10504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:53.506356 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "03aa22a369dd4cd89fae668f0d92e520"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "eb536c3ca32cecc0130e856716ac9671"
server_key_iv: "c74ee94dfa831bceb67c56c6f7c72d58"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.507297 10504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "03aa22a369dd4cd89fae668f0d92e520"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "eb536c3ca32cecc0130e856716ac9671"
server_key_iv: "c74ee94dfa831bceb67c56c6f7c72d58"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.513183 10504 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:53.517828 10518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.519598 10504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260619 01:00:53.519776 10504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "03aa22a369dd4cd89fae668f0d92e520"
format_stamp: "Formatted at 2026-06-19 01:00:53 on dist-test-slave-3rxd"
server_key: "eb536c3ca32cecc0130e856716ac9671"
server_key_iv: "c74ee94dfa831bceb67c56c6f7c72d58"
server_key_version: "encryptionkey@0"
I20260619 01:00:53.519984 10504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:53.525408 10504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:53.526130 10504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:53.526456 10504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:53.527422 10504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:53.528725 10504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:53.528822 10504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.528892 10504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:53.528945 10504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:53.560171 10504 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42983
I20260619 01:00:53.560240 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42983 every 8 connection(s)
I20260619 01:00:53.561518 10504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:53.567274 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10504
I20260619 01:00:53.567444 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:53.567989 388 external_mini_cluster.cc:1428] Setting key c17946168906c6ea3924af4d3c86bc5b
I20260619 01:00:53.569531 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.563998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54621 (local address 127.0.97.62:41711)
0619 01:00:53.564170 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.564174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.565542 (+ 1368us) server_negotiation.cc:407] Connection header received
0619 01:00:53.567221 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.567227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.567291 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.567421 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.568840 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:53.568847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:53.568855 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:53.568870 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:53.568923 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:53.568937 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:53.569065 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:53.569268 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 01:00:53.569330 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:53.570550 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.564529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41711 (local address 127.0.97.3:54621)
0619 01:00:53.565396 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:53.565433 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:53.566973 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:53.567740 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:53.567753 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:53.567867 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:53.568542 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:00:53.568550 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:53.568587 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:00:53.568666 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:53.569037 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:53.569045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:53.569051 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:00:53.569346 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 01:00:53.569596 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":287,"threads_started":1}
I20260619 01:00:53.571734 10631 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41711
I20260619 01:00:53.572007 10631 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:53.572535 10631 heartbeater.cc:511] Master 127.0.97.62:41711 requested a full tablet report, sending...
I20260619 01:00:53.573835 10184 ts_manager.cc:194] Registered new tserver with Master: 03aa22a369dd4cd89fae668f0d92e520 (127.0.97.3:42983)
I20260619 01:00:53.583035 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 01:00:53.588780 10372 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:00:53.587409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49778 (local address 127.0.97.62:41711)
0619 01:00:53.587589 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.587598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.588040 (+ 442us) server_negotiation.cc:407] Connection header received
0619 01:00:53.588141 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.588144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.588187 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.588266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.588556 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49778: BlockingRecv error: recv got EOF from 127.0.0.1:49778 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:53.589596 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10243
I20260619 01:00:53.592698 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.590405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.0.97.1:43247)
0619 01:00:53.590883 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.590896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.590920 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:00:53.590972 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.590977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.591152 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.591276 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.592129 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:53.592135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:53.592145 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:53.592156 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:53.592197 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:53.592213 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:53.592318 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:53.592432 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 01:00:53.592505 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":219,"threads_started":1}
I20260619 01:00:53.593899 10365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:53.675412 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10243
I20260619 01:00:53.692708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10374
I20260619 01:00:53.696197 10642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.693514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.97.2:37683)
0619 01:00:53.693995 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.694001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.694026 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:00:53.694086 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.694092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.694318 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.694442 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.695476 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:53.695483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:53.695495 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:53.695514 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:53.695558 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:53.695572 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:53.695676 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:53.695799 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 01:00:53.695915 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260619 01:00:53.697424 10496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:53.777209 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10374
I20260619 01:00:53.792783 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10504
I20260619 01:00:53.796348 10644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:53.793636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.0.97.3:42983)
0619 01:00:53.794255 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:53.794264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:53.794294 (+ 30us) server_negotiation.cc:407] Connection header received
0619 01:00:53.794356 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:53.794362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:53.794587 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:53.794787 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:53.795648 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:53.795654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:53.795666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:00:53.795677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:53.795714 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:00:53.795728 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:53.795870 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:53.795917 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 01:00:53.796028 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":308,"threads_started":1}
I20260619 01:00:53.797621 10626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:53.877214 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10504
I20260619 01:00:53.893011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10153
I20260619 01:00:53.894196 10214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:53.992868 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10153
2026-06-19T01:00:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2293 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:54 dist-test-slave-3rxd krb5kdc[10649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:54 dist-test-slave-3rxd krb5kdc[10649](info): set up 2 sockets
Jun 19 01:00:54 dist-test-slave-3rxd krb5kdc[10649](info): commencing operation
krb5kdc: starting...
W20260619 01:00:55.075542 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.007s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:55 dist-test-slave-3rxd krb5kdc[10649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:00:55.288213 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42799
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:55.599954 10662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:55.600306 10662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:55.600489 10662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:55.607656 10662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:00:55.607811 10662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:55.607928 10662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:55.608050 10662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:00:55.608150 10662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:00:55.617602 10662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:55.618917 10662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:55.620267 10662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:55.632258 10667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:55.632228 10670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:55.632392 10668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:55.633004 10662 server_base.cc:1034] running on GCE node
I20260619 01:00:55.634069 10662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:55.635476 10662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:55.636700 10662 hybrid_clock.cc:648] HybridClock initialized: now 1781830855636671 us; error 42 us; skew 500 ppm
I20260619 01:00:55.637019 10662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:55 dist-test-slave-3rxd krb5kdc[10649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:55.639940 10662 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:00:55.641245 10662 webserver.cc:460] Webserver started at http://127.0.97.62:34177/ using document root <none> and password file <none>
I20260619 01:00:55.641878 10662 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:55.642025 10662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:55.642330 10662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:55.645144 10662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "e9b753e030034175a00f7457252182cb"
format_stamp: "Formatted at 2026-06-19 01:00:55 on dist-test-slave-3rxd"
server_key: "d382c9b212a77df66bffa2980e195dfd"
server_key_iv: "480bbdd02f02094a6e9f990e1a129eac"
server_key_version: "encryptionkey@0"
I20260619 01:00:55.645884 10662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "e9b753e030034175a00f7457252182cb"
format_stamp: "Formatted at 2026-06-19 01:00:55 on dist-test-slave-3rxd"
server_key: "d382c9b212a77df66bffa2980e195dfd"
server_key_iv: "480bbdd02f02094a6e9f990e1a129eac"
server_key_version: "encryptionkey@0"
I20260619 01:00:55.650764 10662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 01:00:55.653916 10677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:55.655208 10662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:00:55.655371 10662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "e9b753e030034175a00f7457252182cb"
format_stamp: "Formatted at 2026-06-19 01:00:55 on dist-test-slave-3rxd"
server_key: "d382c9b212a77df66bffa2980e195dfd"
server_key_iv: "480bbdd02f02094a6e9f990e1a129eac"
server_key_version: "encryptionkey@0"
I20260619 01:00:55.655544 10662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:55.678241 10662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:55.682052 10662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:55.682319 10662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:55.699980 10662 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42799
I20260619 01:00:55.700014 10728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42799 every 8 connection(s)
I20260619 01:00:55.701222 10662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:00:55.704605 10729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:55.705971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10662
I20260619 01:00:55.706194 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:00:55.707096 388 external_mini_cluster.cc:1428] Setting key f9a8e398388d57dc41d588b2243377d7
I20260619 01:00:55.719162 10729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: Bootstrap starting.
Jun 19 01:00:55 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:55.722141 10729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:55.723518 10729 log.cc:824] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:55.726583 10729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: No bootstrap required, opened a new log
I20260619 01:00:55.730464 10732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:55.709187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58498 (local address 127.0.97.62:42799)
0619 01:00:55.710101 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:55.710117 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:55.710173 (+ 56us) server_negotiation.cc:407] Connection header received
0619 01:00:55.711815 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:55.711842 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:55.712330 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:55.712769 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:55.714134 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:55.716180 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:55.717362 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:55.718378 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:55.718550 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:55.722492 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:55.722503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:55.722527 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:55.722579 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:55.726381 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:55.727063 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:55.727070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:55.727077 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:55.727276 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:55.727588 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:55.727592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:55.727595 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:55.728190 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:55.728500 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:55.728782 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 01:00:55.729192 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":412,"threads_started":1}
I20260619 01:00:55.735260 10729 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260619 01:00:55.735553 10729 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:55.735633 10729 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b753e030034175a00f7457252182cb, State: Initialized, Role: FOLLOWER
I20260619 01:00:55.736124 10729 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260619 01:00:55.736315 10729 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:00:55.736403 10729 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:00:55.736510 10729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:55.738472 10729 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260619 01:00:55.739053 10729 leader_election.cc:302] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b753e030034175a00f7457252182cb; no voters:
I20260619 01:00:55.739703 10729 leader_election.cc:288] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:00:55.739804 10734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:55.740459 10734 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [term 1 LEADER]: Becoming Leader. State: Replica: e9b753e030034175a00f7457252182cb, State: Running, Role: LEADER
I20260619 01:00:55.741004 10734 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } }
I20260619 01:00:55.741593 10729 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:00:55.744007 10735 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b753e030034175a00f7457252182cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } } }
I20260619 01:00:55.743997 10736 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b753e030034175a00f7457252182cb. Latest consensus state: current_term: 1 leader_uuid: "e9b753e030034175a00f7457252182cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b753e030034175a00f7457252182cb" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42799 } } }
I20260619 01:00:55.744375 10736 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:55.744375 10735 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb [sys.catalog]: This master's current role is: LEADER
I20260619 01:00:55.746441 10742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:00:55.751518 10742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:00:55.758703 10742 catalog_manager.cc:1347] Generated new cluster ID: 81f9500426284219881757b5c7bff0fc
I20260619 01:00:55.758822 10742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:00:55.774307 10742 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:00:55.776048 10742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:00:55.792695 10742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: Generated new TSK 0
I20260619 01:00:55.793710 10742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:00:55.853809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:56.171816 10755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:56.172155 10755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:56.172327 10755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:56.179790 10755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:56.179986 10755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:56.180194 10755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:00:56.190032 10755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:56.191359 10755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:56.192619 10755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:56.205535 10761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:56.205480 10760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:56.205837 10763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:56.206537 10755 server_base.cc:1034] running on GCE node
I20260619 01:00:56.207194 10755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:56.208119 10755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:56.209313 10755 hybrid_clock.cc:648] HybridClock initialized: now 1781830856209247 us; error 57 us; skew 500 ppm
I20260619 01:00:56.209626 10755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:56 dist-test-slave-3rxd krb5kdc[10649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:56.212775 10755 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:00:56.214130 10755 webserver.cc:460] Webserver started at http://127.0.97.1:46731/ using document root <none> and password file <none>
I20260619 01:00:56.214886 10755 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:56.215029 10755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:56.215319 10755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:56.218140 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "d0d1a0c31ac443ef806806aa2866b88b"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "646acfb9a2eae1fde2de7a1eff170047"
server_key_iv: "07f2cb7af135eae81ce3d606c365499d"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.218948 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d0d1a0c31ac443ef806806aa2866b88b"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "646acfb9a2eae1fde2de7a1eff170047"
server_key_iv: "07f2cb7af135eae81ce3d606c365499d"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.224161 10755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:00:56.227670 10770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.228979 10755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:00:56.229183 10755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "d0d1a0c31ac443ef806806aa2866b88b"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "646acfb9a2eae1fde2de7a1eff170047"
server_key_iv: "07f2cb7af135eae81ce3d606c365499d"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.229382 10755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:56.245728 10755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:56.250150 10755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:56.250458 10755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:56.251466 10755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:56.252866 10755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:56.252990 10755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.253155 10755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:56.253232 10755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.289053 10755 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40895
I20260619 01:00:56.289078 10882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40895 every 8 connection(s)
I20260619 01:00:56.290387 10755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:00:56.297340 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10755
I20260619 01:00:56.297550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:00:56.298136 388 external_mini_cluster.cc:1428] Setting key 4e40e59388c0cbd7c8f45034d53d2a6d
Jun 19 01:00:56 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:56.310899 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:56.293035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46325 (local address 127.0.97.62:42799)
0619 01:00:56.293444 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:56.293448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:56.294454 (+ 1006us) server_negotiation.cc:407] Connection header received
0619 01:00:56.296281 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:56.296286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:56.296366 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:56.296497 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:56.297992 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.298838 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.300334 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.300737 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.300861 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:56.304701 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:56.304706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:56.304709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:56.304737 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:56.307839 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:56.308490 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:56.308497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:56.308504 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:56.308651 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:56.309023 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:56.309030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:56.309036 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:56.309401 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:56.309544 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:56.310160 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 01:00:56.310398 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":208,"threads_started":1}
I20260619 01:00:56.311528 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:56.293579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.1:46325)
0619 01:00:56.294298 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:56.294344 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:56.296014 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:56.296718 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:56.296726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:56.297207 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:56.297838 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:56.297856 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.298988 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.298993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:56.300188 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:56.300198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.300950 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.300957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:56.301178 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:56.301849 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:56.301870 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:56.304500 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:56.307970 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:56.307977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:56.307988 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:56.308374 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:56.308759 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:56.308762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:56.308764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:56.308928 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:56.309514 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:56.309519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:56.309803 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 01:00:56.310028 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 01:00:56.310367 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":268,"threads_started":1}
I20260619 01:00:56.313063 10883 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260619 01:00:56.313345 10883 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:56.314025 10883 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
I20260619 01:00:56.315986 10694 ts_manager.cc:194] Registered new tserver with Master: d0d1a0c31ac443ef806806aa2866b88b (127.0.97.1:40895)
I20260619 01:00:56.317507 10694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46325
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:00:56.347739 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:56.660730 10889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:56.661118 10889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:56.661332 10889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:56.668883 10889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:56.669030 10889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:56.669209 10889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:00:56.678594 10889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:56.679808 10889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:56.681128 10889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:56.693783 10897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:56.693869 10894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:56.693938 10895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:56.694559 10889 server_base.cc:1034] running on GCE node
I20260619 01:00:56.695158 10889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:56.696053 10889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:56.697274 10889 hybrid_clock.cc:648] HybridClock initialized: now 1781830856697258 us; error 77 us; skew 500 ppm
I20260619 01:00:56.697571 10889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:56 dist-test-slave-3rxd krb5kdc[10649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:56.705694 10889 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:00:56.707278 10889 webserver.cc:460] Webserver started at http://127.0.97.2:43169/ using document root <none> and password file <none>
I20260619 01:00:56.707976 10889 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:56.708101 10889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:56.708408 10889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:56.711252 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "f1036739d23e46d78fb03bfc218b9ec4"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "756b74eae5f6684c405ddc7b12e5d816"
server_key_iv: "115e8495bb4110027bd04396ee82f562"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.711987 10889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f1036739d23e46d78fb03bfc218b9ec4"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "756b74eae5f6684c405ddc7b12e5d816"
server_key_iv: "115e8495bb4110027bd04396ee82f562"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.717057 10889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:00:56.720330 10904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.721560 10889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:00:56.721714 10889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "f1036739d23e46d78fb03bfc218b9ec4"
format_stamp: "Formatted at 2026-06-19 01:00:56 on dist-test-slave-3rxd"
server_key: "756b74eae5f6684c405ddc7b12e5d816"
server_key_iv: "115e8495bb4110027bd04396ee82f562"
server_key_version: "encryptionkey@0"
I20260619 01:00:56.721932 10889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:56.732554 10889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:56.736405 10889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:56.736665 10889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:56.737531 10889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:56.738857 10889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:56.738955 10889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.739116 10889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:56.739174 10889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:56.771601 10889 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43965
I20260619 01:00:56.771647 11016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43965 every 8 connection(s)
I20260619 01:00:56.772889 10889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:00:56.776746 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 10889
I20260619 01:00:56.776932 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:00:56.777410 388 external_mini_cluster.cc:1428] Setting key 5f415ec0cfdc42666a77f65138cff23c
Jun 19 01:00:56 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:56.793385 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:56.775526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36917 (local address 127.0.97.62:42799)
0619 01:00:56.775718 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:56.775722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:56.776908 (+ 1186us) server_negotiation.cc:407] Connection header received
0619 01:00:56.778964 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:56.778969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:56.779054 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:56.779177 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:56.780621 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.781464 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.783074 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.783508 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.783642 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:56.787535 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:56.787540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:56.787543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:56.787575 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:56.790083 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:56.791016 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:56.791019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:56.791021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:56.791141 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:56.791594 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:56.791597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:56.791599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:56.791836 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:56.791937 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:56.792879 (+ 942us) server_negotiation.cc:299] Negotiation successful
0619 01:00:56.793140 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:00:56.794795 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:56.776018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.2:36917)
0619 01:00:56.776731 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:56.776775 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:56.778651 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:56.779385 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:56.779393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:56.779892 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:56.780452 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:56.780467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.781606 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.781614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:56.782926 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:56.782937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:56.783745 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:56.783753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:56.783972 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:56.784652 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:56.784692 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:56.787326 (+ 2634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:56.790226 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:56.790237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:56.790256 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:56.790903 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:56.791245 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:56.791253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:56.791259 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:56.791485 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:56.792018 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:56.792029 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:56.792455 (+ 426us) client_negotiation.cc:769] Sending connection context
0619 01:00:56.792764 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 01:00:56.793312 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":298,"threads_started":1}
I20260619 01:00:56.796264 11017 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260619 01:00:56.796581 11017 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:56.797273 11017 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
I20260619 01:00:56.798769 10694 ts_manager.cc:194] Registered new tserver with Master: f1036739d23e46d78fb03bfc218b9ec4 (127.0.97.2:43965)
I20260619 01:00:56.799926 10694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36917
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:00:56.829872 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:00:57.145128 11022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:00:57.145426 11022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:00:57.145596 11022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:00:57.152555 11022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:00:57.152691 11022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:00:57.152853 11022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:00:57.162094 11022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:00:57.163362 11022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:00:57.164602 11022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:00:57.176668 11028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:57.176788 11027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:00:57.176719 11030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:00:57.177695 11022 server_base.cc:1034] running on GCE node
I20260619 01:00:57.178254 11022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:00:57.179199 11022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:00:57.180414 11022 hybrid_clock.cc:648] HybridClock initialized: now 1781830857180384 us; error 33 us; skew 500 ppm
I20260619 01:00:57.180717 11022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:00:57 dist-test-slave-3rxd krb5kdc[10649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:00:57.183908 11022 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:00:57.185271 11022 webserver.cc:460] Webserver started at http://127.0.97.3:40575/ using document root <none> and password file <none>
I20260619 01:00:57.185979 11022 fs_manager.cc:359] Metadata directory not provided
I20260619 01:00:57.186126 11022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:00:57.186444 11022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:00:57.189441 11022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "9cfac069084a4f9f9571824e3cb11a49"
format_stamp: "Formatted at 2026-06-19 01:00:57 on dist-test-slave-3rxd"
server_key: "e456dd9d9472678f40642155beae7319"
server_key_iv: "508d1528dbdc4e117c0ee4b6764ec44d"
server_key_version: "encryptionkey@0"
I20260619 01:00:57.190191 11022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9cfac069084a4f9f9571824e3cb11a49"
format_stamp: "Formatted at 2026-06-19 01:00:57 on dist-test-slave-3rxd"
server_key: "e456dd9d9472678f40642155beae7319"
server_key_iv: "508d1528dbdc4e117c0ee4b6764ec44d"
server_key_version: "encryptionkey@0"
I20260619 01:00:57.195187 11022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 01:00:57.198418 11037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:57.199702 11022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:00:57.199882 11022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "9cfac069084a4f9f9571824e3cb11a49"
format_stamp: "Formatted at 2026-06-19 01:00:57 on dist-test-slave-3rxd"
server_key: "e456dd9d9472678f40642155beae7319"
server_key_iv: "508d1528dbdc4e117c0ee4b6764ec44d"
server_key_version: "encryptionkey@0"
I20260619 01:00:57.200074 11022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:00:57.212360 11022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:00:57.216187 11022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:00:57.216470 11022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:00:57.217343 11022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:00:57.218571 11022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:00:57.218672 11022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:57.218843 11022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:00:57.218921 11022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:00:57.251629 11022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39261
I20260619 01:00:57.251670 11149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39261 every 8 connection(s)
I20260619 01:00:57.252892 11022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:00:57.261756 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 11022
I20260619 01:00:57.261943 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:00:57.262424 388 external_mini_cluster.cc:1428] Setting key ce7cf7b7be584da56a4e0b7f94845933
Jun 19 01:00:57 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:00:57.273164 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.255239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39015 (local address 127.0.97.62:42799)
0619 01:00:57.255426 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.255431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.256529 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 01:00:57.258270 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.258274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.258335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.258465 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:57.259932 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.260661 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.262673 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.263121 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.263279 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.267188 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:57.267193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:57.267196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:57.267226 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:57.269531 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.270392 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.270399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.270402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.270584 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.271024 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.271032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.271038 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.271396 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:57.271568 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.272586 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.272843 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:00:57.274058 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.255724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42799 (local address 127.0.97.3:39015)
0619 01:00:57.256378 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.256419 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.257985 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.258679 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.258687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.259202 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:00:57.259750 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.259767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.260832 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.260842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.262535 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.262549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.263377 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.263384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.263633 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.264422 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:00:57.264449 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:00:57.266988 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:00:57.269682 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:57.269690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:57.269704 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:57.270273 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:57.270698 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:00:57.270703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:00:57.270726 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:00:57.270914 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:00:57.271639 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:00:57.271646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:00:57.272072 (+ 426us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.272464 (+ 392us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.272982 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260619 01:00:57.275425 11150 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42799
I20260619 01:00:57.275769 11150 heartbeater.cc:464] Registering TS with master...
I20260619 01:00:57.276671 11150 heartbeater.cc:511] Master 127.0.97.62:42799 requested a full tablet report, sending...
I20260619 01:00:57.277983 10694 ts_manager.cc:194] Registered new tserver with Master: 9cfac069084a4f9f9571824e3cb11a49 (127.0.97.3:39261)
I20260619 01:00:57.278781 10694 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39015
I20260619 01:00:57.289429 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:00:57.305763 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.293902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.0.97.62:42799)
0619 01:00:57.294061 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.294066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.294571 (+ 505us) server_negotiation.cc:407] Connection header received
0619 01:00:57.294695 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.294698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.294814 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.294893 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:57.296154 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.296832 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.297963 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.298335 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.298452 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.300567 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:57.300572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:57.300575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:57.300621 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:57.303426 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.304106 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.304110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.304112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.304250 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.304558 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.304560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.304562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.304893 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:57.305065 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.305394 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.305534 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:00:57.309693 10694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:00:57.316332 10694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:00:57.320340 10883 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
I20260619 01:00:57.351013 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.328339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60406 (local address 127.0.97.3:39261)
0619 01:00:57.341071 (+ 12732us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.341084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.342211 (+ 1127us) server_negotiation.cc:407] Connection header received
0619 01:00:57.342359 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.342369 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.342678 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.342929 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:57.344086 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.345970 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.348522 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.349497 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.349615 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.349826 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.350483 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.350631 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12514,"thread_start_us":12879,"threads_started":1}
I20260619 01:00:57.351048 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.328055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39261 (local address 127.0.0.1:60406)
0619 01:00:57.341817 (+ 13762us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.341853 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.342027 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.343278 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.343289 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.343329 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:57.343842 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.343856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.346111 (+ 2255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.346122 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.348382 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.348392 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.350020 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.350032 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.350273 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.350309 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.350439 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.350612 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13535,"mutex_wait_us":16,"thread_start_us":15267,"threads_started":1}
I20260619 01:00:57.352566 11162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.328464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40895 (local address 127.0.0.1:55058)
0619 01:00:57.343095 (+ 14631us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.343116 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.343241 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.345727 (+ 2486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.345732 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.345755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:57.346166 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.346183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.347518 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.347525 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.349846 (+ 2321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.349872 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.351683 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.351690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.351871 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.351941 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.352034 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.352145 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14469,"mutex_wait_us":116,"thread_start_us":12992,"threads_started":1}
I20260619 01:00:57.352650 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43965 (local address 127.0.0.1:52302)
0619 01:00:57.344347 (+ 15401us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.344364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.344536 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.345201 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.345207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.345245 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:57.345585 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.345599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.347716 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.347726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.350090 (+ 2364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.350102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.351868 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.351876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.352055 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.352086 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.352151 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.352280 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15288,"mutex_wait_us":57,"thread_start_us":15161,"threads_started":1}
I20260619 01:00:57.353674 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.329557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.0.97.2:43965)
0619 01:00:57.343912 (+ 14355us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.343924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.344474 (+ 550us) server_negotiation.cc:407] Connection header received
0619 01:00:57.344701 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.344712 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.344996 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.345773 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:57.346030 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.347584 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.350236 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.351637 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.352976 (+ 1339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.353138 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.353202 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.353361 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14189,"thread_start_us":14082,"threads_started":1}
I20260619 01:00:57.355309 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.328944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55058 (local address 127.0.97.1:40895)
0619 01:00:57.345160 (+ 16216us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.345171 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.345195 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:00:57.345276 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.345284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.345528 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.345750 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:57.346350 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.347394 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.350024 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.351456 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.351598 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.351857 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.352083 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.352236 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16042,"thread_start_us":15886,"threads_started":1}
I20260619 01:00:57.355561 11085 tablet_service.cc:1467] Processing CreateTablet for tablet 21bbb35f300542b5a42ff797b82377c0 (DEFAULT_TABLE table=test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:57.356787 11085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21bbb35f300542b5a42ff797b82377c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:57.356890 10952 tablet_service.cc:1467] Processing CreateTablet for tablet 21bbb35f300542b5a42ff797b82377c0 (DEFAULT_TABLE table=test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:57.357896 10952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21bbb35f300542b5a42ff797b82377c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:57.359205 10818 tablet_service.cc:1467] Processing CreateTablet for tablet 21bbb35f300542b5a42ff797b82377c0 (DEFAULT_TABLE table=test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:00:57.360419 10818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21bbb35f300542b5a42ff797b82377c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:00:57.369213 11168 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Bootstrap starting.
I20260619 01:00:57.370806 11169 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Bootstrap starting.
I20260619 01:00:57.373262 11168 tablet_bootstrap.cc:654] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:57.373462 11170 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Bootstrap starting.
I20260619 01:00:57.374517 11168 log.cc:824] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:57.375098 11169 tablet_bootstrap.cc:654] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:57.376153 11169 log.cc:824] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:57.376533 11170 tablet_bootstrap.cc:654] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Neither blocks nor log segments found. Creating new log.
I20260619 01:00:57.377447 11170 log.cc:824] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:00:57.378604 11168 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: No bootstrap required, opened a new log
I20260619 01:00:57.378961 11168 ts_tablet_manager.cc:1397] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 01:00:57.379808 11170 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: No bootstrap required, opened a new log
I20260619 01:00:57.380048 11170 ts_tablet_manager.cc:1397] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 01:00:57.384871 11169 tablet_bootstrap.cc:492] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: No bootstrap required, opened a new log
I20260619 01:00:57.385105 11169 ts_tablet_manager.cc:1397] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260619 01:00:57.387377 11170 raft_consensus.cc:348] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.387707 11170 raft_consensus.cc:374] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:57.387833 11170 raft_consensus.cc:729] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d1a0c31ac443ef806806aa2866b88b, State: Initialized, Role: FOLLOWER
I20260619 01:00:57.388422 11170 consensus_queue.cc:260] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.390039 11170 ts_tablet_manager.cc:1428] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260619 01:00:57.390224 11168 raft_consensus.cc:348] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.390614 11168 raft_consensus.cc:374] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:57.390695 11168 raft_consensus.cc:729] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1036739d23e46d78fb03bfc218b9ec4, State: Initialized, Role: FOLLOWER
I20260619 01:00:57.391254 11168 consensus_queue.cc:260] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.392871 11168 ts_tablet_manager.cc:1428] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 01:00:57.393015 11017 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
I20260619 01:00:57.395255 11169 raft_consensus.cc:348] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.395659 11169 raft_consensus.cc:374] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:00:57.395761 11169 raft_consensus.cc:729] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cfac069084a4f9f9571824e3cb11a49, State: Initialized, Role: FOLLOWER
I20260619 01:00:57.396309 11169 consensus_queue.cc:260] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.397281 11150 heartbeater.cc:503] Master 127.0.97.62:42799 was elected leader, sending a full tablet report...
I20260619 01:00:57.397781 11169 ts_tablet_manager.cc:1428] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 01:00:57.462860 11175 raft_consensus.cc:482] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:00:57.463131 11175 raft_consensus.cc:504] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.464610 11175 leader_election.cc:288] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cfac069084a4f9f9571824e3cb11a49 (127.0.97.3:39261), d0d1a0c31ac443ef806806aa2866b88b (127.0.97.1:40895)
I20260619 01:00:57.471801 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.465034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44471 (local address 127.0.97.3:39261)
0619 01:00:57.465229 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.465235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.465986 (+ 751us) server_negotiation.cc:407] Connection header received
0619 01:00:57.466158 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.466163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.466235 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.466337 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:57.467114 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.468037 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.469992 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.470763 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.470871 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.470945 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.471380 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.471519 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:00:57.471968 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.464892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40895 (local address 127.0.97.2:46459)
0619 01:00:57.465803 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.465824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.465989 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.466371 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.466376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.466426 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:57.466955 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.466963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.468203 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.468207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.469786 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.469798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.471135 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.471139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.471336 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.471369 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.471457 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.471569 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":74,"spinlock_wait_cycles":25856,"thread_start_us":584,"threads_started":1}
I20260619 01:00:57.471995 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.464992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39261 (local address 127.0.97.2:44471)
0619 01:00:57.465875 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0619 01:00:57.465893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:00:57.466033 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:00:57.466371 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:00:57.466376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:00:57.466401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:00:57.466947 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.466960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.468174 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.468180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.469791 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:00:57.469803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.470975 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.470984 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:00:57.471221 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.471255 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:00:57.471326 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 01:00:57.471415 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":82,"spinlock_wait_cycles":21760,"thread_start_us":418,"threads_started":1}
I20260619 01:00:57.471848 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.465096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46459 (local address 127.0.97.1:40895)
0619 01:00:57.465309 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.465317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.465954 (+ 637us) server_negotiation.cc:407] Connection header received
0619 01:00:57.466153 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.466159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.466235 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.466336 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:00:57.467164 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.468064 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.469942 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.470927 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.471065 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.471166 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.471497 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.471630 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:00:57.472998 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bbb35f300542b5a42ff797b82377c0" candidate_uuid: "f1036739d23e46d78fb03bfc218b9ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfac069084a4f9f9571824e3cb11a49" is_pre_election: true
I20260619 01:00:57.473024 10838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bbb35f300542b5a42ff797b82377c0" candidate_uuid: "f1036739d23e46d78fb03bfc218b9ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d1a0c31ac443ef806806aa2866b88b" is_pre_election: true
I20260619 01:00:57.473590 11105 raft_consensus.cc:2454] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1036739d23e46d78fb03bfc218b9ec4 in term 0.
I20260619 01:00:57.473590 10838 raft_consensus.cc:2454] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1036739d23e46d78fb03bfc218b9ec4 in term 0.
I20260619 01:00:57.474304 10908 leader_election.cc:302] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9cfac069084a4f9f9571824e3cb11a49, d0d1a0c31ac443ef806806aa2866b88b, f1036739d23e46d78fb03bfc218b9ec4; no voters:
I20260619 01:00:57.474692 11175 raft_consensus.cc:2781] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:00:57.474896 11175 raft_consensus.cc:482] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:00:57.474995 11175 raft_consensus.cc:3037] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:57.477103 11175 raft_consensus.cc:504] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.477710 11175 leader_election.cc:288] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [CANDIDATE]: Term 1 election: Requested vote from peers 9cfac069084a4f9f9571824e3cb11a49 (127.0.97.3:39261), d0d1a0c31ac443ef806806aa2866b88b (127.0.97.1:40895)
I20260619 01:00:57.478107 11105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bbb35f300542b5a42ff797b82377c0" candidate_uuid: "f1036739d23e46d78fb03bfc218b9ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cfac069084a4f9f9571824e3cb11a49"
I20260619 01:00:57.478106 10838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21bbb35f300542b5a42ff797b82377c0" candidate_uuid: "f1036739d23e46d78fb03bfc218b9ec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d1a0c31ac443ef806806aa2866b88b"
I20260619 01:00:57.478315 10838 raft_consensus.cc:3037] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:57.478299 11105 raft_consensus.cc:3037] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:00:57.480463 11105 raft_consensus.cc:2454] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1036739d23e46d78fb03bfc218b9ec4 in term 1.
I20260619 01:00:57.480468 10838 raft_consensus.cc:2454] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1036739d23e46d78fb03bfc218b9ec4 in term 1.
I20260619 01:00:57.480839 10908 leader_election.cc:302] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cfac069084a4f9f9571824e3cb11a49, f1036739d23e46d78fb03bfc218b9ec4; no voters:
I20260619 01:00:57.481117 11175 raft_consensus.cc:2781] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:00:57.481513 11175 raft_consensus.cc:686] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 1 LEADER]: Becoming Leader. State: Replica: f1036739d23e46d78fb03bfc218b9ec4, State: Running, Role: LEADER
I20260619 01:00:57.481961 11175 consensus_queue.cc:237] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } }
I20260619 01:00:57.486301 10693 catalog_manager.cc:5495] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1036739d23e46d78fb03bfc218b9ec4 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f1036739d23e46d78fb03bfc218b9ec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1036739d23e46d78fb03bfc218b9ec4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43965 } health_report { overall_health: HEALTHY } } }
W20260619 01:00:57.504439 11151 tablet.cc:2369] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:57.509286 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.503748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.0.97.2:43965)
0619 01:00:57.503962 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.503969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.504187 (+ 218us) server_negotiation.cc:407] Connection header received
0619 01:00:57.504409 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.504414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.504485 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.504640 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:00:57.505046 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.505749 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.507137 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.507539 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.507673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.508140 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:00:57.508691 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:00:57.508857 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.508911 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.508985 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:00:57.518055 11105 raft_consensus.cc:1261] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 1 FOLLOWER]: Refusing update from remote peer f1036739d23e46d78fb03bfc218b9ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:57.519068 11179 consensus_queue.cc:1035] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cfac069084a4f9f9571824e3cb11a49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:00:57.519275 10838 raft_consensus.cc:1261] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 1 FOLLOWER]: Refusing update from remote peer f1036739d23e46d78fb03bfc218b9ec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:00:57.520053 11179 consensus_queue.cc:1035] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d1a0c31ac443ef806806aa2866b88b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:00:57.524761 11018 tablet.cc:2369] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:00:57.533460 11184 mvcc.cc:205] Tried to move back new op lower bound from 7298379192383197184 to 7298379192250335232. Current Snapshot: MvccSnapshot[applied={T|T < 7298379192383197184}]
I20260619 01:00:57.535303 11183 mvcc.cc:205] Tried to move back new op lower bound from 7298379192383197184 to 7298379192250335232. Current Snapshot: MvccSnapshot[applied={T|T < 7298379192383197184}]
I20260619 01:00:57.549069 10694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:57.549371 10694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:00:57.552834 10694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9b753e030034175a00f7457252182cb: Sending DeleteTablet for 3 replicas of tablet 21bbb35f300542b5a42ff797b82377c0
I20260619 01:00:57.553921 11085 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bbb35f300542b5a42ff797b82377c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60406
I20260619 01:00:57.553913 10818 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bbb35f300542b5a42ff797b82377c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55058
I20260619 01:00:57.554018 10952 tablet_service.cc:1514] Processing DeleteTablet for tablet 21bbb35f300542b5a42ff797b82377c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:00:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52302
I20260619 01:00:57.557590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10755
I20260619 01:00:57.566287 11192 tablet_replica.cc:330] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: stopping tablet replica
I20260619 01:00:57.566505 11194 tablet_replica.cc:330] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: stopping tablet replica
I20260619 01:00:57.567041 11194 raft_consensus.cc:2229] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:57.567764 11194 raft_consensus.cc:2258] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:57.567824 11192 raft_consensus.cc:2229] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:00:57.569070 11192 raft_consensus.cc:2258] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:57.570240 11194 ts_tablet_manager.cc:1905] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:57.571708 11192 ts_tablet_manager.cc:1905] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:57.576051 11193 tablet_replica.cc:330] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: stopping tablet replica
I20260619 01:00:57.576459 11193 raft_consensus.cc:2229] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:00:57.576907 11193 raft_consensus.cc:2258] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:00:57.577976 11194 ts_tablet_manager.cc:1918] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:57.578181 11194 log.cc:1196] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/21bbb35f300542b5a42ff797b82377c0
I20260619 01:00:57.578809 11194 ts_tablet_manager.cc:1939] T 21bbb35f300542b5a42ff797b82377c0 P 9cfac069084a4f9f9571824e3cb11a49: Deleting consensus metadata
I20260619 01:00:57.580659 10681 catalog_manager.cc:4841] TS 9cfac069084a4f9f9571824e3cb11a49 (127.0.97.3:39261): tablet 21bbb35f300542b5a42ff797b82377c0 (table test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]) successfully deleted
I20260619 01:00:57.581494 11192 ts_tablet_manager.cc:1918] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:57.581662 11192 log.cc:1196] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/21bbb35f300542b5a42ff797b82377c0
Jun 19 01:00:57 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:00:57.582259 11192 ts_tablet_manager.cc:1939] T 21bbb35f300542b5a42ff797b82377c0 P f1036739d23e46d78fb03bfc218b9ec4: Deleting consensus metadata
I20260619 01:00:57.583871 10678 catalog_manager.cc:4841] TS f1036739d23e46d78fb03bfc218b9ec4 (127.0.97.2:43965): tablet 21bbb35f300542b5a42ff797b82377c0 (table test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]) successfully deleted
I20260619 01:00:57.584816 11193 ts_tablet_manager.cc:1905] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:00:57.590397 11193 ts_tablet_manager.cc:1918] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:00:57.590574 11193 log.cc:1196] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/21bbb35f300542b5a42ff797b82377c0
I20260619 01:00:57.590756 11165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.559834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.97.1:40895)
0619 01:00:57.560007 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.560013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.571621 (+ 11608us) server_negotiation.cc:407] Connection header received
0619 01:00:57.572076 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.572082 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.572189 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.572269 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:57.573935 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.575103 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.576997 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.577650 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.577763 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.583109 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:57.583126 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:57.583144 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:57.583189 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:57.587336 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.588187 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.588197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.588201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.588357 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.588772 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.588778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.588781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.589285 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:57.589579 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.589872 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.590105 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:00:57.591163 11193 ts_tablet_manager.cc:1939] T 21bbb35f300542b5a42ff797b82377c0 P d0d1a0c31ac443ef806806aa2866b88b: Deleting consensus metadata
I20260619 01:00:57.591396 10878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:57.700954 10679 catalog_manager.cc:4841] TS d0d1a0c31ac443ef806806aa2866b88b (127.0.97.1:40895): tablet 21bbb35f300542b5a42ff797b82377c0 (table test-table [id=02c788bdcf674ca2ba6013e67cb36dfc]) successfully deleted
I20260619 01:00:57.702778 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10755
I20260619 01:00:57.720789 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10889
Jun 19 01:00:57 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:00:57.736850 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.721536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.0.97.2:43965)
0619 01:00:57.721737 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.721756 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.721811 (+ 55us) server_negotiation.cc:407] Connection header received
0619 01:00:57.722016 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.722021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.722095 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.722212 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:57.723488 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.724346 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.725733 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.726335 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.726464 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.730906 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:57.730921 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:57.730937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:57.730985 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:57.733997 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.734836 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.734846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.734850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.735048 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.735440 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.735443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.735444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.735840 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:57.736068 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.736390 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.736541 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:00:57.737579 11012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:57.847227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10889
I20260619 01:00:57.866113 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 11022
Jun 19 01:00:57 dist-test-slave-3rxd krb5kdc[10649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:00:57.880419 11163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:00:57.866859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.0.97.3:39261)
0619 01:00:57.867043 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 01:00:57.867050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:00:57.867145 (+ 95us) server_negotiation.cc:407] Connection header received
0619 01:00:57.867360 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:00:57.867364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:00:57.867426 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:00:57.867525 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:00:57.868731 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.869443 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.870682 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:00:57.871127 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:00:57.871274 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:00:57.875324 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:00:57.875337 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:00:57.875353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:00:57.875388 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:00:57.877734 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.878353 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.878369 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.878371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.878506 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:00:57.878934 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:00:57.878942 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:00:57.878947 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:00:57.879297 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:00:57.879586 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0619 01:00:57.879898 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 01:00:57.880111 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:00:57.881002 11145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:57.967271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 11022
I20260619 01:00:57.985781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 10662
I20260619 01:00:57.986960 10724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:00:58.091799 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 10662
2026-06-19T01:00:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4100 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:00:58 dist-test-slave-3rxd krb5kdc[11202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:00:58 dist-test-slave-3rxd krb5kdc[11202](info): set up 2 sockets
Jun 19 01:00:58 dist-test-slave-3rxd krb5kdc[11202](info): commencing operation
krb5kdc: starting...
W20260619 01:00:59.203418 388 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:00:59 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:00:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:00:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 01:00:59.501178 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2ewLDB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/postgres -l logfile start
2026-06-19 01:01:01.903 UTC [11232] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 01:01:01.903 UTC [11232] LOG: listening on IPv4 address "127.0.97.20", port 37005
2026-06-19 01:01:01.906 UTC [11232] LOG: listening on Unix socket "/tmp/.s.PGSQL.37005"
2026-06-19 01:01:01.928 UTC [11235] LOG: database system was shut down at 2026-06-19 01:01:00 UTC
2026-06-19 01:01:01.932 UTC [11232] LOG: database system is ready to accept connections
I20260619 01:01:02.915341 388 mini_postgres.cc:96] Postgres bound to 37005
2026-06-19 01:01:02.932 UTC [11243] FATAL: database "slave" does not exist
127.0.97.20:37005 - accepting connections
I20260619 01:01:02.932966 388 mini_ranger.cc:161] Starting Ranger...
I20260619 01:01:02.958235 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 01:01:03.427503 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 01:01:03.427639 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin
2026-06-19 01:01:03,713 [I] DB FLAVOR :POSTGRES
2026-06-19 01:01:03,714 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:01:03,714 [I] Checking connection..
2026-06-19 01:01:03,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 01:01:03,960 [I] Checking connection passed.
2026-06-19 01:01:03,960 [I] --------- Verifying version history table ---------
2026-06-19 01:01:03,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:04,218 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:01:04,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:04,475 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:01:04,476 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 01:01:04,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 01:01:04.714 UTC [11307] WARNING: there is no transaction in progress
2026-06-19 01:01:04,720 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 01:01:04,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:04,951 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 01:01:04,951 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:01:04,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 01:01:05,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 01:01:05,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3rxd.c.gcp-upstream.internal','N') ;"
2026-06-19 01:01:05,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:01:05,901 [I] Table x_portal_user does not exist in database ranger
2026-06-19 01:01:05,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:01:06,151 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 01:01:06,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:01:06,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:01:06,609 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 01:01:06,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 01:01:07.522 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.537 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.553 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.567 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.582 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.601 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.664 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.670 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.683 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.693 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.703 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.717 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.726 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.733 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.739 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:07.745 UTC [11415] WARNING: there is no transaction in progress
2026-06-19 01:01:08,317 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 01:01:08,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 01:01:08,573 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 01:01:08,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:01:08,826 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 01:01:08,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:01:09,051 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 01:01:09,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:01:09,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:01:09,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3rxd.c.gcp-upstream.internal';"
2026-06-19 01:01:09,749 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 01:01:09,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 01:01:09,998 [I] DB_PATCHES have already been applied
I20260619 01:01:10.004118 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2ewLDB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 01:01:10.004204 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 01:01:10.018286 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:38847
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:01:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 01:01:11.137236 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.204s user 0.000s sys 0.036s
Jun 19, 2026 1:01:31 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:01:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 1:01:31 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:01:31 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:01:32 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 01:01:32.730432 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 01:01:32 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830892, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 01:01:32.898203 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 01:01:32.917232 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 01:01:33.717244 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 01:01:33.717423 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms
I20260619 01:01:33.717562 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:37005
2026-06-19 01:01:34,121 [I] DB FLAVOR :POSTGRES
2026-06-19 01:01:34,121 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:01:34,121 [I] Checking connection
2026-06-19 01:01:34,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 01:01:34,344 [I] connection success
2026-06-19 01:01:34,345 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 01:01:34,345 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 01:01:34,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 01:01:34,606 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 01:01:34,607 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:01:34,607 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 01:01:34,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:37005/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 01:01:34,857 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 01:01:35.315436 388 mini_ranger_kms.cc:310] Created kms service
I20260619 01:01:35.523088 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 01:01:35.566753 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 01:01:35 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830892, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:01:35.953711 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 01:01:35.954020 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 01:01:35.954088 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 01:01:35.974468 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11595
I20260619 01:01:35.974650 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51899
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:01:37 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 01:01:38.302852 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.405s user 0.134s sys 0.282s
I20260619 01:01:38.303098 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 01:01:38.303303 388 mini_ranger_kms.cc:401] 127.0.97.20:51899/kms/v1/keys
Jun 19 01:01:38 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 01:01:38 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:01:39 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830898, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:01:41 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:01:41.507118 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38809
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38809
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51899/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:01:41.845968 11646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:01:41.846235 11646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:01:41.846392 11646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:01:41.853868 11646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:01:41.853986 11646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:01:41.854097 11646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:01:41.854194 11646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:01:41.854277 11646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:01:41.864174 11646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51899/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38809
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:01:41.865526 11646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:01:41.866835 11646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:01:41.879060 11654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:41.879096 11652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:41.879120 11651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:01:41.879516 11646 server_base.cc:1034] running on GCE node
I20260619 01:01:41.880473 11646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:01:41.881727 11646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:01:41.882918 11646 hybrid_clock.cc:648] HybridClock initialized: now 1781830901882865 us; error 44 us; skew 500 ppm
I20260619 01:01:41.883193 11646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:01:41 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:01:41.886053 11646 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:01:41.887444 11646 webserver.cc:460] Webserver started at http://127.0.97.62:42165/ using document root <none> and password file <none>
I20260619 01:01:41.888024 11646 fs_manager.cc:359] Metadata directory not provided
I20260619 01:01:41.888111 11646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:01:41.888343 11646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:01:41 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:01:41.945333 11646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "afff0d7dde954bdf8dc7070297b11f2d"
format_stamp: "Formatted at 2026-06-19 01:01:41 on dist-test-slave-3rxd"
server_key: "e367969154531c07ad5209322512f4ff"
server_key_iv: "0a9cd5f75aff1c6e71f658526c8d8c93"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:41.946391 11646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "afff0d7dde954bdf8dc7070297b11f2d"
format_stamp: "Formatted at 2026-06-19 01:01:41 on dist-test-slave-3rxd"
server_key: "e367969154531c07ad5209322512f4ff"
server_key_iv: "0a9cd5f75aff1c6e71f658526c8d8c93"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:41.953541 11646 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260619 01:01:41.973351 11661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:41.975107 11646 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:01:41.975286 11646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "afff0d7dde954bdf8dc7070297b11f2d"
format_stamp: "Formatted at 2026-06-19 01:01:41 on dist-test-slave-3rxd"
server_key: "e367969154531c07ad5209322512f4ff"
server_key_iv: "0a9cd5f75aff1c6e71f658526c8d8c93"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:41.975543 11646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:01:42.000833 11646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:01:42.006084 11646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:01:42.006390 11646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:01:42.025820 11712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38809 every 8 connection(s)
I20260619 01:01:42.025830 11646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38809
I20260619 01:01:42.027518 11646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:01:42.029384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 11646
I20260619 01:01:42.029592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:01:42.030386 388 external_mini_cluster.cc:1428] Setting key c94dbcbb7e79362d877823180f38ded5
I20260619 01:01:42.033479 11713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:01:42.045588 11713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: Bootstrap starting.
Jun 19 01:01:42 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:01:42.049961 11713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: Neither blocks nor log segments found. Creating new log.
I20260619 01:01:42.051360 11713 log.cc:824] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:01:42.054776 11713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: No bootstrap required, opened a new log
I20260619 01:01:42.055749 11716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:42.032744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.97.62:38809)
0619 01:01:42.033418 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:42.033434 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:42.033493 (+ 59us) server_negotiation.cc:407] Connection header received
0619 01:01:42.035676 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:42.035700 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:42.036157 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:42.036580 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:42.038043 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:42.040137 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:42.041894 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:42.043055 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:42.043231 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:42.046829 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:42.046840 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:42.046859 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:42.046911 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:42.051219 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:42.051952 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:42.051965 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:42.051978 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:42.052162 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:42.052578 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:42.052582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:42.052584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:42.053267 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:42.053535 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:42.053926 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 01:01:42.054314 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":309,"threads_started":1}
I20260619 01:01:42.063405 11713 raft_consensus.cc:348] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260619 01:01:42.063722 11713 raft_consensus.cc:374] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:01:42.063794 11713 raft_consensus.cc:729] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afff0d7dde954bdf8dc7070297b11f2d, State: Initialized, Role: FOLLOWER
I20260619 01:01:42.064427 11713 consensus_queue.cc:260] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260619 01:01:42.064638 11713 raft_consensus.cc:388] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:01:42.064726 11713 raft_consensus.cc:482] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:01:42.064841 11713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:01:42.067301 11713 raft_consensus.cc:504] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260619 01:01:42.067735 11713 leader_election.cc:302] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afff0d7dde954bdf8dc7070297b11f2d; no voters:
I20260619 01:01:42.068250 11713 leader_election.cc:288] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:01:42.068389 11718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:01:42.069072 11718 raft_consensus.cc:686] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [term 1 LEADER]: Becoming Leader. State: Replica: afff0d7dde954bdf8dc7070297b11f2d, State: Running, Role: LEADER
I20260619 01:01:42.069636 11718 consensus_queue.cc:237] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } }
I20260619 01:01:42.070322 11713 sys_catalog.cc:564] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:01:42.072474 11720 sys_catalog.cc:455] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader afff0d7dde954bdf8dc7070297b11f2d. Latest consensus state: current_term: 1 leader_uuid: "afff0d7dde954bdf8dc7070297b11f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } } }
I20260619 01:01:42.072578 11719 sys_catalog.cc:455] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afff0d7dde954bdf8dc7070297b11f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afff0d7dde954bdf8dc7070297b11f2d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38809 } } }
I20260619 01:01:42.072885 11719 sys_catalog.cc:458] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [sys.catalog]: This master's current role is: LEADER
I20260619 01:01:42.072885 11720 sys_catalog.cc:458] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d [sys.catalog]: This master's current role is: LEADER
I20260619 01:01:42.075310 11724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:01:42.081391 11724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:01:42.086993 11713 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 01:01:42.100718 11724 catalog_manager.cc:1347] Generated new cluster ID: b997a0fda380470a954877cc125ed416
I20260619 01:01:42.100888 11724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:01:42.120107 11724 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:01:42.121997 11724 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:01:42.140327 11724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: Generated new TSK 0
I20260619 01:01:42.141407 11724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:01:42.878508 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 01:01:43.303512 11758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:01:43.303793 11758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:01:43.303978 11758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:01:43.311842 11758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:01:43.311978 11758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:01:43.312156 11758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:01:43.322542 11758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:01:43.324311 11758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:01:43.325716 11758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:01:43.346457 11768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:43.346573 11769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:43.358278 11772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:01:43.358819 11758 server_base.cc:1034] running on GCE node
I20260619 01:01:43.359527 11758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:01:43.360553 11758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:01:43.361768 11758 hybrid_clock.cc:648] HybridClock initialized: now 1781830903361703 us; error 60 us; skew 500 ppm
I20260619 01:01:43.362102 11758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:01:43.365455 11758 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:01:43.371984 11758 webserver.cc:460] Webserver started at http://127.0.97.1:45773/ using document root <none> and password file <none>
I20260619 01:01:43.372707 11758 fs_manager.cc:359] Metadata directory not provided
I20260619 01:01:43.372854 11758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:01:43.373168 11758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:01:43.392908 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "df473df380824faf96ca59afb317d27e"
format_stamp: "Formatted at 2026-06-19 01:01:43 on dist-test-slave-3rxd"
server_key: "65f508a388cd595df202dcbe49523e6a"
server_key_iv: "117c76ff40c94c71841e01edf75172bb"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:43.394047 11758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "df473df380824faf96ca59afb317d27e"
format_stamp: "Formatted at 2026-06-19 01:01:43 on dist-test-slave-3rxd"
server_key: "65f508a388cd595df202dcbe49523e6a"
server_key_iv: "117c76ff40c94c71841e01edf75172bb"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:43.412345 11758 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.008s sys 0.008s
I20260619 01:01:43.448444 11778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:43.450570 11758 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.006s sys 0.013s
I20260619 01:01:43.450875 11758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "df473df380824faf96ca59afb317d27e"
format_stamp: "Formatted at 2026-06-19 01:01:43 on dist-test-slave-3rxd"
server_key: "65f508a388cd595df202dcbe49523e6a"
server_key_iv: "117c76ff40c94c71841e01edf75172bb"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:43.451254 11758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:01:43.488672 11758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:01:43.496354 11758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:01:43.497138 11758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:01:43.498348 11758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:01:43.504130 11758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:01:43.504282 11758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:43.504451 11758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:01:43.504525 11758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:43.609328 11758 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37899
I20260619 01:01:43.609375 11890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37899 every 8 connection(s)
I20260619 01:01:43.611232 11758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:01:43.615010 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 11758
I20260619 01:01:43.615211 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:01:43.616003 388 external_mini_cluster.cc:1428] Setting key 4fdf2289a2e77377d828f69463781440
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:01:43.661223 11894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:43.625284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58867 (local address 127.0.97.62:38809)
0619 01:01:43.627452 (+ 2168us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:43.627458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:43.628505 (+ 1047us) server_negotiation.cc:407] Connection header received
0619 01:01:43.630809 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:43.630815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:43.630894 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:43.630980 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:43.633413 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:43.635006 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:43.637483 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:43.638111 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:43.638210 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:43.642695 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:43.642702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:43.642733 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:43.642772 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:43.648284 (+ 5512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:43.658490 (+ 10206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:43.658497 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:43.658500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:43.658672 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:43.659226 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:43.659231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:43.659234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:43.659576 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:43.659690 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:43.660671 (+ 981us) server_negotiation.cc:299] Negotiation successful
0619 01:01:43.660883 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2030,"thread_start_us":1912,"threads_started":1}
I20260619 01:01:43.663416 11893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:43.624760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.1:58867)
0619 01:01:43.628030 (+ 3270us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:43.628081 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:43.630460 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:43.631655 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:43.631670 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:43.632275 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:01:43.633143 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:43.633168 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:43.635199 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:43.635204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:43.637305 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:43.637320 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:43.638558 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:43.638566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:43.638857 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:43.639631 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:01:43.639664 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:01:43.642450 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:01:43.657719 (+ 15269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:43.657730 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:43.657750 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:43.658346 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:43.658812 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:43.658816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:43.658820 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:43.659088 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:43.659787 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:01:43.659796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:01:43.660231 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 01:01:43.661491 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0619 01:01:43.662033 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2959,"thread_start_us":2728,"threads_started":1}
I20260619 01:01:43.665078 11891 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260619 01:01:43.665381 11891 heartbeater.cc:464] Registering TS with master...
I20260619 01:01:43.666167 11891 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260619 01:01:43.668697 11678 ts_manager.cc:194] Registered new tserver with Master: df473df380824faf96ca59afb317d27e (127.0.97.1:37899)
I20260619 01:01:43.670835 11678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58867
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:01:43.709854 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:01:43 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:01:44.188089 11897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:01:44.188447 11897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:01:44.188619 11897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:01:44.196163 11897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:01:44.196312 11897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:01:44.196511 11897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:01:44.206971 11897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:01:44.208334 11897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:01:44.209786 11897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:01:44.245332 11907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:44.246127 11904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:01:44.248346 11897 server_base.cc:1034] running on GCE node
W20260619 01:01:44.248466 11912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:01:44.249150 11897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:01:44.250113 11897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:01:44.251336 11897 hybrid_clock.cc:648] HybridClock initialized: now 1781830904251273 us; error 57 us; skew 500 ppm
I20260619 01:01:44.252040 11897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:01:44.255379 11897 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:01:44.257143 11897 webserver.cc:460] Webserver started at http://127.0.97.2:36589/ using document root <none> and password file <none>
I20260619 01:01:44.258044 11897 fs_manager.cc:359] Metadata directory not provided
I20260619 01:01:44.258167 11897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:01:44.258507 11897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:01:44.278486 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "1abcd6c8383d41ae9939bdfecdee7a7f"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "d15a7dd27a2088abd8d8a354a67cbce2"
server_key_iv: "af261c7232748e0cbec3f478ad730159"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.279598 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1abcd6c8383d41ae9939bdfecdee7a7f"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "d15a7dd27a2088abd8d8a354a67cbce2"
server_key_iv: "af261c7232748e0cbec3f478ad730159"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.286504 11897 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 01:01:44.306032 11921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:44.307906 11897 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:01:44.308185 11897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "1abcd6c8383d41ae9939bdfecdee7a7f"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "d15a7dd27a2088abd8d8a354a67cbce2"
server_key_iv: "af261c7232748e0cbec3f478ad730159"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.308523 11897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:01:44.343139 11897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:01:44.348790 11897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:01:44.349207 11897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:01:44.350453 11897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:01:44.352298 11897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:01:44.352447 11897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:01:44.352619 11897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:01:44.352713 11897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:44.395642 11897 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33069
I20260619 01:01:44.395656 12033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33069 every 8 connection(s)
I20260619 01:01:44.397006 11897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:01:44.401875 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 11897
I20260619 01:01:44.402048 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:01:44.402622 388 external_mini_cluster.cc:1428] Setting key fb7057f8500aa281f2f2897e8c5696c8
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:01:44.421530 12037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:44.400911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43223 (local address 127.0.97.62:38809)
0619 01:01:44.401394 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:44.401400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:44.401420 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:01:44.403940 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:44.403946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:44.404051 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:44.404179 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:44.406073 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.407317 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.409911 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.410684 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.410834 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:44.415287 (+ 4453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:44.415292 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:44.415295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:44.415331 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:44.418416 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.419222 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.419229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.419233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.419360 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.419804 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.419809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.419813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.420095 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:44.420249 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:44.420918 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 01:01:44.421134 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":915,"threads_started":1}
I20260619 01:01:44.422567 12036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:44.400407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.2:43223)
0619 01:01:44.401255 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:44.401305 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:44.403625 (+ 2320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:44.404451 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:44.404464 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:44.405062 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:01:44.405860 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:44.405902 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.407499 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.407508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:44.409713 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:44.409731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.411138 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.411146 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:44.411469 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:44.412295 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:01:44.412330 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:01:44.415010 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:01:44.418569 (+ 3559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:44.418578 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:44.418596 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:44.419102 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:44.419477 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:44.419483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:44.419486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:44.419688 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:44.420224 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:01:44.420231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:01:44.420570 (+ 339us) client_negotiation.cc:769] Sending connection context
0619 01:01:44.420833 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 01:01:44.421360 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":572,"threads_started":1}
I20260619 01:01:44.424099 12034 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260619 01:01:44.424424 12034 heartbeater.cc:464] Registering TS with master...
I20260619 01:01:44.425130 12034 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260619 01:01:44.426532 11678 ts_manager.cc:194] Registered new tserver with Master: 1abcd6c8383d41ae9939bdfecdee7a7f (127.0.97.2:33069)
I20260619 01:01:44.427420 11678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:43223
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:01:44.465571 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51899/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:01:44.674320 11891 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
W20260619 01:01:44.799544 12040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:01:44.799803 12040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:01:44.799981 12040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:01:44.806870 12040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:01:44.807015 12040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:01:44.807214 12040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:01:44.816962 12040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51899/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38809
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:01:44.818209 12040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:01:44.819555 12040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:01:44.832744 12048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:44.832805 12046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:01:44.832841 12045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:01:44.833633 12040 server_base.cc:1034] running on GCE node
I20260619 01:01:44.834239 12040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:01:44.835156 12040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:01:44.836385 12040 hybrid_clock.cc:648] HybridClock initialized: now 1781830904836338 us; error 55 us; skew 500 ppm
I20260619 01:01:44.836681 12040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:01:44.840068 12040 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:01:44.841341 12040 webserver.cc:460] Webserver started at http://127.0.97.3:40301/ using document root <none> and password file <none>
I20260619 01:01:44.841966 12040 fs_manager.cc:359] Metadata directory not provided
I20260619 01:01:44.842087 12040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:01:44.842391 12040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:01:44.859061 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "a5c1afe28a1342b38b8737425e7fcc98"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "a0a0e30719b0d03c97bc1e813740df5f"
server_key_iv: "3c2e569f93eb7d8f0177130b67d6e018"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.859886 12040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a5c1afe28a1342b38b8737425e7fcc98"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "a0a0e30719b0d03c97bc1e813740df5f"
server_key_iv: "3c2e569f93eb7d8f0177130b67d6e018"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.865435 12040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:01:44.881218 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:44.882889 12040 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:01:44.883075 12040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "a5c1afe28a1342b38b8737425e7fcc98"
format_stamp: "Formatted at 2026-06-19 01:01:44 on dist-test-slave-3rxd"
server_key: "a0a0e30719b0d03c97bc1e813740df5f"
server_key_iv: "3c2e569f93eb7d8f0177130b67d6e018"
server_key_version: "kuduclusterkey@0"
I20260619 01:01:44.883304 12040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:01:44.898029 12040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:01:44.901600 12040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:01:44.901889 12040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:01:44.902829 12040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:01:44.904094 12040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:01:44.904196 12040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:44.904340 12040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:01:44.904409 12040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:01:44.938289 12040 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33001
I20260619 01:01:44.938308 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33001 every 8 connection(s)
I20260619 01:01:44.939688 12040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:01:44.949682 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 12040
I20260619 01:01:44.949865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:01:44.950455 388 external_mini_cluster.cc:1428] Setting key 8a8ac92d339afa16bd9634ab1d6af575
Jun 19 01:01:44 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:01:44.959005 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:44.942069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44693 (local address 127.0.97.62:38809)
0619 01:01:44.942532 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:44.942538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:44.943501 (+ 963us) server_negotiation.cc:407] Connection header received
0619 01:01:44.945152 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:44.945156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:44.945229 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:44.945324 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:44.946644 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.947402 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.948879 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.949264 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.949351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:44.953420 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:44.953426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:44.953430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:44.953470 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:44.955845 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.956526 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.956529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.956531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.956652 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.957109 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.957112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.957114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.957360 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:44.957565 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:44.958106 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 01:01:44.958264 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":191,"threads_started":1}
I20260619 01:01:44.959493 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:44.942600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38809 (local address 127.0.97.3:44693)
0619 01:01:44.943341 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:44.943383 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:44.944915 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:44.945544 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:44.945552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:44.945936 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:01:44.946503 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:44.946520 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.947547 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.947551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:44.948754 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:44.948765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.949486 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.949495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:44.949795 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:44.950496 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:01:44.950527 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:01:44.953094 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:01:44.955973 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:44.955979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:44.955989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:44.956435 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:44.956759 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:01:44.956762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:01:44.956764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:01:44.956974 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:01:44.957544 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:01:44.957550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:01:44.957812 (+ 262us) client_negotiation.cc:769] Sending connection context
0619 01:01:44.958035 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 01:01:44.958363 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":287,"threads_started":1}
I20260619 01:01:44.960678 12168 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38809
I20260619 01:01:44.960976 12168 heartbeater.cc:464] Registering TS with master...
I20260619 01:01:44.961637 12168 heartbeater.cc:511] Master 127.0.97.62:38809 requested a full tablet report, sending...
I20260619 01:01:44.963012 11678 ts_manager.cc:194] Registered new tserver with Master: a5c1afe28a1342b38b8737425e7fcc98 (127.0.97.3:33001)
I20260619 01:01:44.963831 11678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44693
I20260619 01:01:44.965579 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:01:44.982348 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:44.970046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.97.62:38809)
0619 01:01:44.970192 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:44.970196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:44.970660 (+ 464us) server_negotiation.cc:407] Connection header received
0619 01:01:44.970898 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:44.970903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:44.970976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:44.971083 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:44.972389 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.973071 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.974493 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:44.974970 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:44.975091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:44.977370 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:44.977375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:44.977378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:44.977411 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:44.979984 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.980671 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.980674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.980677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.980830 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:44.981141 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:44.981144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:44.981146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:44.981448 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:44.981618 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:44.981893 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 01:01:44.982041 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:01:44.985703 11678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:01:44.993139 11678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:01:45.016330 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.005319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55022 (local address 127.0.97.3:33001)
0619 01:01:45.005984 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.005993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.006027 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:01:45.006232 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.006240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.006526 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.006735 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:01:45.007719 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.009803 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.013731 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.015249 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.015382 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.015617 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.015859 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.016012 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":357,"threads_started":1}
I20260619 01:01:45.016353 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.005090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33001 (local address 127.0.0.1:55022)
0619 01:01:45.005842 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:45.005893 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:45.006039 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:45.007041 (+ 1002us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:45.007049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:45.007089 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:01:45.007473 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.007491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.009999 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.010007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.013549 (+ 3542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.013564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.015425 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.015431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.015642 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.015694 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 01:01:45.015822 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 01:01:45.015992 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":423,"thread_start_us":302,"threads_started":1}
I20260619 01:01:45.017455 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.006227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33069 (local address 127.0.0.1:39592)
0619 01:01:45.006756 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:45.006779 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:45.006914 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:45.008061 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:45.008069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:45.008102 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:01:45.008613 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.008628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.011345 (+ 2717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.011354 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.013884 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.013896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.016618 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.016624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.016795 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.016824 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:01:45.016899 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 01:01:45.017010 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":3,"thread_start_us":254,"threads_started":1}
I20260619 01:01:45.018170 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.006507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.97.2:33069)
0619 01:01:45.007398 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.007406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.007432 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:01:45.007537 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.007548 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.007812 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.008854 (+ 1042us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:01:45.009144 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.011145 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.014057 (+ 2912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.016154 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.016286 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.016471 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.017675 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.017835 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":638,"threads_started":1}
I20260619 01:01:45.021389 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.007177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35016 (local address 127.0.97.1:37899)
0619 01:01:45.013003 (+ 5826us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.013012 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.013680 (+ 668us) server_negotiation.cc:407] Connection header received
0619 01:01:45.013995 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.014003 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.014216 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.014383 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:01:45.015243 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.017013 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.019359 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.020299 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.020460 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.020675 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.020897 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.021074 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5677,"thread_start_us":5460,"threads_started":1}
I20260619 01:01:45.020823 12103 tablet_service.cc:1467] Processing CreateTablet for tablet 69145c48bcd747bca9e685524e77cc44 (DEFAULT_TABLE table=test-table [id=e456c5e835734994baf6db19232b08b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:01:45.021483 11969 tablet_service.cc:1467] Processing CreateTablet for tablet 69145c48bcd747bca9e685524e77cc44 (DEFAULT_TABLE table=test-table [id=e456c5e835734994baf6db19232b08b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:01:45.022107 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.007453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37899 (local address 127.0.0.1:35016)
0619 01:01:45.013556 (+ 6103us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:45.013572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:45.013769 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:45.014650 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:45.014654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:45.014677 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:01:45.015055 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.015066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.017163 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.017167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.019209 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.019221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.020512 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.020519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.020676 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.020719 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 01:01:45.021776 (+ 1057us) client_negotiation.cc:240] Negotiation successful
0619 01:01:45.021861 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5977,"thread_start_us":5847,"threads_started":1}
I20260619 01:01:45.022296 12103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69145c48bcd747bca9e685524e77cc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:01:45.022810 11969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69145c48bcd747bca9e685524e77cc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:01:45.025094 11826 tablet_service.cc:1467] Processing CreateTablet for tablet 69145c48bcd747bca9e685524e77cc44 (DEFAULT_TABLE table=test-table [id=e456c5e835734994baf6db19232b08b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:01:45.026399 11826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69145c48bcd747bca9e685524e77cc44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:01:45.043537 12187 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Bootstrap starting.
I20260619 01:01:45.044225 12188 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Bootstrap starting.
I20260619 01:01:45.047343 12189 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Bootstrap starting.
I20260619 01:01:45.048705 12187 tablet_bootstrap.cc:654] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Neither blocks nor log segments found. Creating new log.
I20260619 01:01:45.049620 12188 tablet_bootstrap.cc:654] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Neither blocks nor log segments found. Creating new log.
I20260619 01:01:45.049947 12187 log.cc:824] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Log is configured to *not* fsync() on all Append() calls
I20260619 01:01:45.050848 12188 log.cc:824] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:01:45.052991 12187 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: No bootstrap required, opened a new log
I20260619 01:01:45.053246 12187 ts_tablet_manager.cc:1397] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 01:01:45.053948 12188 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: No bootstrap required, opened a new log
I20260619 01:01:45.054186 12188 ts_tablet_manager.cc:1397] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 01:01:45.054301 12189 tablet_bootstrap.cc:654] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Neither blocks nor log segments found. Creating new log.
I20260619 01:01:45.055490 12189 log.cc:824] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Log is configured to *not* fsync() on all Append() calls
I20260619 01:01:45.058229 12189 tablet_bootstrap.cc:492] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: No bootstrap required, opened a new log
I20260619 01:01:45.058463 12189 ts_tablet_manager.cc:1397] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20260619 01:01:45.061331 12187 raft_consensus.cc:348] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.061694 12187 raft_consensus.cc:374] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:01:45.061807 12187 raft_consensus.cc:729] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5c1afe28a1342b38b8737425e7fcc98, State: Initialized, Role: FOLLOWER
I20260619 01:01:45.062377 12187 consensus_queue.cc:260] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.064301 12188 raft_consensus.cc:348] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.064671 12188 raft_consensus.cc:374] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:01:45.064764 12188 raft_consensus.cc:729] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abcd6c8383d41ae9939bdfecdee7a7f, State: Initialized, Role: FOLLOWER
I20260619 01:01:45.065281 12188 consensus_queue.cc:260] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.066457 12034 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
I20260619 01:01:45.067013 12188 ts_tablet_manager.cc:1428] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 01:01:45.068174 12189 raft_consensus.cc:348] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.068562 12189 raft_consensus.cc:374] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:01:45.068665 12189 raft_consensus.cc:729] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df473df380824faf96ca59afb317d27e, State: Initialized, Role: FOLLOWER
I20260619 01:01:45.069216 12189 consensus_queue.cc:260] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.070622 12168 heartbeater.cc:503] Master 127.0.97.62:38809 was elected leader, sending a full tablet report...
I20260619 01:01:45.071447 12187 ts_tablet_manager.cc:1428] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Time spent starting tablet: real 0.018s user 0.001s sys 0.019s
I20260619 01:01:45.072912 12189 ts_tablet_manager.cc:1428] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
W20260619 01:01:45.119671 11892 tablet.cc:2369] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:01:45.122517 12195 raft_consensus.cc:482] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:01:45.122807 12195 raft_consensus.cc:504] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.124280 12195 leader_election.cc:288] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5c1afe28a1342b38b8737425e7fcc98 (127.0.97.3:33001), 1abcd6c8383d41ae9939bdfecdee7a7f (127.0.97.2:33069)
I20260619 01:01:45.131465 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.124653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49793 (local address 127.0.97.3:33001)
0619 01:01:45.124857 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.124863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.125323 (+ 460us) server_negotiation.cc:407] Connection header received
0619 01:01:45.125559 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.125566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.125643 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.125733 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:01:45.126360 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.127319 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.129412 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.130577 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.130703 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.130827 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.131090 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.131209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:01:45.131557 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.124540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33001 (local address 127.0.97.1:49793)
0619 01:01:45.125228 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:45.125246 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:45.125375 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:45.125772 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:45.125779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:45.125807 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:01:45.126188 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.126201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.127439 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.127444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.129259 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.129271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.130779 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.130785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.130936 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.130958 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:01:45.131070 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:01:45.131198 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260619 01:01:45.131808 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.125499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56473 (local address 127.0.97.2:33069)
0619 01:01:45.125676 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.125683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.126035 (+ 352us) server_negotiation.cc:407] Connection header received
0619 01:01:45.126143 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.126148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.126220 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.126301 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:01:45.126908 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.127714 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.129943 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.130916 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.131068 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.131161 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.131437 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.131555 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:01:45.131932 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.125323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33069 (local address 127.0.97.1:56473)
0619 01:01:45.125941 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 01:01:45.125954 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:01:45.126046 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:01:45.126336 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:01:45.126341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:01:45.126363 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:01:45.126768 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.126780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.127879 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.127882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.129796 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:01:45.129811 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.131129 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.131135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:01:45.131288 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.131310 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:01:45.131414 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:01:45.131547 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":344,"threads_started":1}
I20260619 01:01:45.132687 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69145c48bcd747bca9e685524e77cc44" candidate_uuid: "df473df380824faf96ca59afb317d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1afe28a1342b38b8737425e7fcc98" is_pre_election: true
I20260619 01:01:45.133008 11989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69145c48bcd747bca9e685524e77cc44" candidate_uuid: "df473df380824faf96ca59afb317d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" is_pre_election: true
I20260619 01:01:45.133170 12123 raft_consensus.cc:2454] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df473df380824faf96ca59afb317d27e in term 0.
I20260619 01:01:45.133459 11989 raft_consensus.cc:2454] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df473df380824faf96ca59afb317d27e in term 0.
I20260619 01:01:45.133869 11781 leader_election.cc:302] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c1afe28a1342b38b8737425e7fcc98, df473df380824faf96ca59afb317d27e; no voters:
I20260619 01:01:45.134286 12195 raft_consensus.cc:2781] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:01:45.134435 12195 raft_consensus.cc:482] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:01:45.134557 12195 raft_consensus.cc:3037] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:01:45.136716 12195 raft_consensus.cc:504] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.137262 12195 leader_election.cc:288] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [CANDIDATE]: Term 1 election: Requested vote from peers a5c1afe28a1342b38b8737425e7fcc98 (127.0.97.3:33001), 1abcd6c8383d41ae9939bdfecdee7a7f (127.0.97.2:33069)
I20260619 01:01:45.137645 12123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69145c48bcd747bca9e685524e77cc44" candidate_uuid: "df473df380824faf96ca59afb317d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5c1afe28a1342b38b8737425e7fcc98"
I20260619 01:01:45.137691 11989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69145c48bcd747bca9e685524e77cc44" candidate_uuid: "df473df380824faf96ca59afb317d27e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f"
I20260619 01:01:45.137816 12123 raft_consensus.cc:3037] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:01:45.137907 11989 raft_consensus.cc:3037] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:01:45.139995 12123 raft_consensus.cc:2454] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df473df380824faf96ca59afb317d27e in term 1.
I20260619 01:01:45.140319 11781 leader_election.cc:302] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5c1afe28a1342b38b8737425e7fcc98, df473df380824faf96ca59afb317d27e; no voters:
I20260619 01:01:45.140566 12195 raft_consensus.cc:2781] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:01:45.140992 12195 raft_consensus.cc:686] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 1 LEADER]: Becoming Leader. State: Replica: df473df380824faf96ca59afb317d27e, State: Running, Role: LEADER
I20260619 01:01:45.141044 11989 raft_consensus.cc:2454] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df473df380824faf96ca59afb317d27e in term 1.
I20260619 01:01:45.141563 12195 consensus_queue.cc:237] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } }
I20260619 01:01:45.145815 11676 catalog_manager.cc:5495] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e reported cstate change: term changed from 0 to 1, leader changed from <none> to df473df380824faf96ca59afb317d27e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "df473df380824faf96ca59afb317d27e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df473df380824faf96ca59afb317d27e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37899 } health_report { overall_health: HEALTHY } } }
W20260619 01:01:45.148964 12035 tablet.cc:2369] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:01:45.165907 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.160774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.97.1:37899)
0619 01:01:45.160975 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.160981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.160999 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:01:45.161183 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.161188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.161276 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.161429 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:01:45.161785 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.162502 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.163955 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.164330 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.164468 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.164929 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:01:45.165379 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:01:45.165547 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.165595 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.165667 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:01:45.174054 11989 raft_consensus.cc:1261] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 1 FOLLOWER]: Refusing update from remote peer df473df380824faf96ca59afb317d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:01:45.174115 12123 raft_consensus.cc:1261] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 1 FOLLOWER]: Refusing update from remote peer df473df380824faf96ca59afb317d27e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:01:45.175035 12198 consensus_queue.cc:1035] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abcd6c8383d41ae9939bdfecdee7a7f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:01:45.175396 12195 consensus_queue.cc:1035] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5c1afe28a1342b38b8737425e7fcc98" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:01:45.185931 12201 mvcc.cc:205] Tried to move back new op lower bound from 7298379387584139264 to 7298379387463434240. Current Snapshot: MvccSnapshot[applied={T|T < 7298379387584139264}]
I20260619 01:01:45.191098 12203 mvcc.cc:205] Tried to move back new op lower bound from 7298379387584139264 to 7298379387463434240. Current Snapshot: MvccSnapshot[applied={T|T < 7298379387584139264}]
I20260619 01:01:45.202111 11678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:01:45.202379 11678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:01:45.205670 11678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afff0d7dde954bdf8dc7070297b11f2d: Sending DeleteTablet for 3 replicas of tablet 69145c48bcd747bca9e685524e77cc44
I20260619 01:01:45.206643 12103 tablet_service.cc:1514] Processing DeleteTablet for tablet 69145c48bcd747bca9e685524e77cc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:01:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55022
I20260619 01:01:45.207075 11969 tablet_service.cc:1514] Processing DeleteTablet for tablet 69145c48bcd747bca9e685524e77cc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:01:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39592
I20260619 01:01:45.207099 11826 tablet_service.cc:1514] Processing DeleteTablet for tablet 69145c48bcd747bca9e685524e77cc44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:01:45 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35016
I20260619 01:01:45.208520 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 11758
I20260619 01:01:45.219583 12212 tablet_replica.cc:330] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: stopping tablet replica
I20260619 01:01:45.220091 12212 raft_consensus.cc:2229] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:01:45.220758 12212 raft_consensus.cc:2258] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:01:45.221457 12213 tablet_replica.cc:330] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: stopping tablet replica
I20260619 01:01:45.221835 12213 raft_consensus.cc:2229] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:01:45.222337 12213 raft_consensus.cc:2258] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:01:45.223172 12211 tablet_replica.cc:330] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: stopping tablet replica
I20260619 01:01:45.223569 12211 raft_consensus.cc:2229] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:01:45.224045 12211 raft_consensus.cc:2258] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:01:45.224583 12212 ts_tablet_manager.cc:1905] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:01:45.225853 12213 ts_tablet_manager.cc:1905] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:01:45.225868 12211 ts_tablet_manager.cc:1905] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 01:01:45 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:01:45.232831 12211 ts_tablet_manager.cc:1918] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:01:45.233031 12211 log.cc:1196] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/69145c48bcd747bca9e685524e77cc44
I20260619 01:01:45.234016 12211 ts_tablet_manager.cc:1939] T 69145c48bcd747bca9e685524e77cc44 P a5c1afe28a1342b38b8737425e7fcc98: Deleting consensus metadata
I20260619 01:01:45.236120 11664 catalog_manager.cc:4841] TS a5c1afe28a1342b38b8737425e7fcc98 (127.0.97.3:33001): tablet 69145c48bcd747bca9e685524e77cc44 (table test-table [id=e456c5e835734994baf6db19232b08b2]) successfully deleted
I20260619 01:01:45.238168 12213 ts_tablet_manager.cc:1918] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:01:45.238343 12212 ts_tablet_manager.cc:1918] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:01:45.238395 12213 log.cc:1196] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/69145c48bcd747bca9e685524e77cc44
I20260619 01:01:45.238546 12212 log.cc:1196] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/69145c48bcd747bca9e685524e77cc44
I20260619 01:01:45.239037 12213 ts_tablet_manager.cc:1939] T 69145c48bcd747bca9e685524e77cc44 P 1abcd6c8383d41ae9939bdfecdee7a7f: Deleting consensus metadata
I20260619 01:01:45.239202 12212 ts_tablet_manager.cc:1939] T 69145c48bcd747bca9e685524e77cc44 P df473df380824faf96ca59afb317d27e: Deleting consensus metadata
I20260619 01:01:45.240793 11664 catalog_manager.cc:4841] TS 1abcd6c8383d41ae9939bdfecdee7a7f (127.0.97.2:33069): tablet 69145c48bcd747bca9e685524e77cc44 (table test-table [id=e456c5e835734994baf6db19232b08b2]) successfully deleted
I20260619 01:01:45.241066 11664 catalog_manager.cc:4841] TS df473df380824faf96ca59afb317d27e (127.0.97.1:37899): tablet 69145c48bcd747bca9e685524e77cc44 (table test-table [id=e456c5e835734994baf6db19232b08b2]) successfully deleted
I20260619 01:01:45.241592 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.210587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.0.97.1:37899)
0619 01:01:45.210816 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.210825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.220708 (+ 9883us) server_negotiation.cc:407] Connection header received
0619 01:01:45.221045 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.221053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.221149 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.221234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:45.223849 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.224973 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.226860 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.227589 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.227696 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.233495 (+ 5799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:45.233509 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:45.233526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:45.233567 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:45.238072 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.239023 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.239035 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.239038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.239222 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.239697 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.239702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.239705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.240165 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:45.240442 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.241077 (+ 635us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.241325 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:01:45.242975 11886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:01:45.339444 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 11758
I20260619 01:01:45.357857 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 11897
Jun 19 01:01:45 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:01:45.371631 12184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.358589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.0.97.2:33069)
0619 01:01:45.358810 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.358815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.358831 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:01:45.358949 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.358953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.359044 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.359126 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:45.360311 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.361013 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.362206 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.362572 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.362663 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.366688 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:45.366701 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:45.366766 (+ 65us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:45.366809 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:45.369332 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.369926 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.369934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.369936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.370058 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.370355 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.370357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.370359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.370748 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:45.371035 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.371220 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.371382 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 01:01:45.372282 12029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:01:45.459383 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 11897
I20260619 01:01:45.475664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 12040
Jun 19 01:01:45 dist-test-slave-3rxd krb5kdc[11202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:01:45.490849 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:01:45.476415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.97.3:33001)
0619 01:01:45.476584 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:01:45.476591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:01:45.476663 (+ 72us) server_negotiation.cc:407] Connection header received
0619 01:01:45.476893 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:01:45.476898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:01:45.477016 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:01:45.477120 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:01:45.478512 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.479338 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.480712 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:01:45.481173 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:01:45.481351 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:01:45.485511 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:01:45.485523 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:01:45.485539 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:01:45.485567 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:01:45.488361 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.488962 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.488969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.488971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.489094 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:01:45.489415 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:01:45.489418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:01:45.489419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:01:45.489755 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:01:45.489961 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 01:01:45.490234 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 01:01:45.490378 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:01:45.491492 12163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:01:45.581467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 12040
I20260619 01:01:45.597455 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 11646
I20260619 01:01:45.598513 11708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:01:45.707607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 11646
I20260619 01:01:45.728201 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 01:01:46.072919 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 01:01:46.073169 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 01:01:46.418810 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 01:01:46.419 UTC [11232] LOG: received smart shutdown request
2026-06-19 01:01:46.422 UTC [11232] LOG: background worker "logical replication launcher" (PID 11241) exited with exit code 1
2026-06-19 01:01:46.422 UTC [11236] LOG: shutting down
2026-06-19 01:01:46.570 UTC [11232] LOG: database system is shut down
2026-06-19T01:01:46Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48597 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:01:46 dist-test-slave-3rxd krb5kdc[12227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:01:46 dist-test-slave-3rxd krb5kdc[12227](info): set up 2 sockets
Jun 19 01:01:46 dist-test-slave-3rxd krb5kdc[12227](info): commencing operation
krb5kdc: starting...
W20260619 01:01:47.798951 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:01:47 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:01:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:01:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260619 01:01:48.101764 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task2ewLDB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/postgres -l logfile start
2026-06-19 01:01:50.506 UTC [12257] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 01:01:50.506 UTC [12257] LOG: listening on IPv4 address "127.0.97.20", port 54673
2026-06-19 01:01:50.509 UTC [12257] LOG: listening on Unix socket "/tmp/.s.PGSQL.54673"
2026-06-19 01:01:50.532 UTC [12260] LOG: database system was shut down at 2026-06-19 01:01:49 UTC
2026-06-19 01:01:50.536 UTC [12257] LOG: database system is ready to accept connections
I20260619 01:01:51.502739 388 mini_postgres.cc:96] Postgres bound to 54673
2026-06-19 01:01:51.522 UTC [12268] FATAL: database "slave" does not exist
127.0.97.20:54673 - accepting connections
I20260619 01:01:51.523469 388 mini_ranger.cc:161] Starting Ranger...
I20260619 01:01:51.552659 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 01:01:52.020572 388 mini_ranger.cc:87] Created ranger Postgres database
I20260619 01:01:52.020689 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin
2026-06-19 01:01:52,316 [I] DB FLAVOR :POSTGRES
2026-06-19 01:01:52,316 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:01:52,316 [I] Checking connection..
2026-06-19 01:01:52,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 01:01:52,553 [I] Checking connection passed.
2026-06-19 01:01:52,553 [I] --------- Verifying version history table ---------
2026-06-19 01:01:52,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:52,799 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:01:52,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:53,048 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 01:01:53,048 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 01:01:53,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 01:01:53.299 UTC [12332] WARNING: there is no transaction in progress
2026-06-19 01:01:53,305 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 01:01:53,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 01:01:53,542 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 01:01:53,542 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:01:53,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 01:01:53,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 01:01:54,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3rxd.c.gcp-upstream.internal','N') ;"
2026-06-19 01:01:54,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:01:54,496 [I] Table x_portal_user does not exist in database ranger
2026-06-19 01:01:54,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:01:54,736 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 01:01:54,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:01:55,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:01:55,259 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 01:01:55,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 01:01:56.235 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.251 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.267 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.282 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.298 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.317 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.372 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.378 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.392 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.402 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.412 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.426 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.436 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.443 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.451 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:56.457 UTC [12440] WARNING: there is no transaction in progress
2026-06-19 01:01:57,096 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 01:01:57,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 01:01:57,356 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 01:01:57,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 01:01:57,612 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 01:01:57,612 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 01:01:57,846 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 01:01:57,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 01:01:58,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 01:01:58,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3rxd.c.gcp-upstream.internal';"
2026-06-19 01:01:58,582 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 01:01:58,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 01:01:58,825 [I] DB_PATCHES have already been applied
I20260619 01:01:58.831238 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task2ewLDB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 01:01:58.831318 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260619 01:01:58.845248 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:52083
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 1:01:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 01:01:59.979425 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.456s user 0.001s sys 0.039s
Jun 19, 2026 1:02:18 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 1:02:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 1:02:18 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 1:02:18 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 1:02:19 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 01:02:20.141287 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 01:02:20 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830940, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 01:02:20.298414 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 01:02:20.317543 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 01:02:21.073861 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 01:02:21.073951 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms
I20260619 01:02:21.074002 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:54673
2026-06-19 01:02:21,421 [I] DB FLAVOR :POSTGRES
2026-06-19 01:02:21,422 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 01:02:21,422 [I] Checking connection
2026-06-19 01:02:21,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 01:02:21,669 [I] connection success
2026-06-19 01:02:21,669 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 01:02:21,670 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 01:02:21,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 01:02:21,902 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 01:02:21,903 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 01:02:21,903 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 01:02:21,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:54673/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 01:02:22,141 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 01:02:22.585490 388 mini_ranger_kms.cc:310] Created kms service
I20260619 01:02:22.790046 388 mini_ranger_kms.cc:326] Created kudu user
I20260619 01:02:22.832095 388 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 01:02:22 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830940, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:23.248636 388 mini_ranger_kms.cc:384] Added ranger policy
I20260619 01:02:23.248946 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms:/tmp/dist-test-task2ewLDB/build/asan/bin/postgresql.jar:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task2ewLDB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 01:02:23.249009 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260619 01:02:23.265542 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12620
I20260619 01:02:23.265646 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:59063
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 1:02:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 01:02:25.506582 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.208s user 0.073s sys 0.289s
I20260619 01:02:25.506832 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 01:02:25.507016 388 mini_ranger_kms.cc:401] 127.0.97.20:59063/kms/v1/keys
Jun 19 01:02:25 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 01:02:26 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:02:26 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Jun 19 01:02:28 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:02:28.668769 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41135
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41135
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:29.005402 12671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:29.005726 12671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:29.005887 12671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:29.012985 12671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:29.013123 12671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:29.013231 12671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:29.013324 12671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:29.013419 12671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:29.023352 12671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:59063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41135
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:29.024580 12671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:29.025842 12671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:29.037468 12679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:29.037508 12677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:29.037539 12676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:29.038564 12671 server_base.cc:1034] running on GCE node
I20260619 01:02:29.039363 12671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:29.040603 12671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:29.041824 12671 hybrid_clock.cc:648] HybridClock initialized: now 1781830949041790 us; error 44 us; skew 500 ppm
I20260619 01:02:29.042244 12671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:29 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:29.045432 12671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:29.046886 12671 webserver.cc:460] Webserver started at http://127.0.97.62:44619/ using document root <none> and password file <none>
I20260619 01:02:29.047463 12671 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:29.047556 12671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:29.047797 12671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:02:29 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:29.096303 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "a8dce4de2b674690b4254d15c10684f8"
format_stamp: "Formatted at 2026-06-19 01:02:29 on dist-test-slave-3rxd"
server_key: "407a87e1b2d5d17ac930e1cad738a09d"
server_key_iv: "0339c53b8d64f029a50c2ab04309603c"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:29.097221 12671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "a8dce4de2b674690b4254d15c10684f8"
format_stamp: "Formatted at 2026-06-19 01:02:29 on dist-test-slave-3rxd"
server_key: "407a87e1b2d5d17ac930e1cad738a09d"
server_key_iv: "0339c53b8d64f029a50c2ab04309603c"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:29.102984 12671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 01:02:29.123152 12686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:29.124941 12671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:02:29.125156 12671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "a8dce4de2b674690b4254d15c10684f8"
format_stamp: "Formatted at 2026-06-19 01:02:29 on dist-test-slave-3rxd"
server_key: "407a87e1b2d5d17ac930e1cad738a09d"
server_key_iv: "0339c53b8d64f029a50c2ab04309603c"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:29.125399 12671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:29.151746 12671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:29.155908 12671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:29.156251 12671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:29.176616 12671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41135
I20260619 01:02:29.176651 12737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41135 every 8 connection(s)
I20260619 01:02:29.177995 12671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:29.182399 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 12671
I20260619 01:02:29.182649 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:02:29.183013 12738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:29.183388 388 external_mini_cluster.cc:1428] Setting key 6a50adcb98fffb50e31acbe0fd128ab7
I20260619 01:02:29.196808 12738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Bootstrap starting.
I20260619 01:02:29.200500 12738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:29.201924 12738 log.cc:824] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:29.204980 12738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: No bootstrap required, opened a new log
Jun 19 01:02:29 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:29.215161 12738 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:29.215541 12738 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:29.215659 12738 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8dce4de2b674690b4254d15c10684f8, State: Initialized, Role: FOLLOWER
I20260619 01:02:29.216295 12738 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:29.216590 12738 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:29.216734 12738 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:29.216907 12738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:29.219501 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:29.185703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.97.62:41135)
0619 01:02:29.197587 (+ 11884us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:29.197604 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:29.198227 (+ 623us) server_negotiation.cc:407] Connection header received
0619 01:02:29.200116 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:29.200138 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:29.200531 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:29.200931 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:29.202644 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:29.204198 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:29.205942 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:29.206608 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:29.206826 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:29.211259 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:29.211273 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:29.211293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:29.211334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:29.215036 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:29.215718 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:29.215729 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:29.215738 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:29.215909 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:29.216321 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:29.216327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:29.216330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:29.217056 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:29.217337 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:29.217692 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 01:02:29.218120 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11621,"thread_start_us":11355,"threads_started":1}
I20260619 01:02:29.220172 12738 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:29.220763 12738 leader_election.cc:302] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8dce4de2b674690b4254d15c10684f8; no voters:
I20260619 01:02:29.221434 12738 leader_election.cc:288] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:02:29.221551 12743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:29.222728 12743 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 LEADER]: Becoming Leader. State: Replica: a8dce4de2b674690b4254d15c10684f8, State: Running, Role: LEADER
I20260619 01:02:29.223291 12743 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:29.223781 12738 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:29.227783 12745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8dce4de2b674690b4254d15c10684f8. Latest consensus state: current_term: 1 leader_uuid: "a8dce4de2b674690b4254d15c10684f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } } }
I20260619 01:02:29.228641 12745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:29.227792 12744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8dce4de2b674690b4254d15c10684f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } } }
I20260619 01:02:29.230979 12744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:29.233297 12753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:29.237931 12753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:29.239933 12738 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 01:02:29.254796 12753 catalog_manager.cc:1347] Generated new cluster ID: 0342432b2ece490c883454898b602d9e
I20260619 01:02:29.255007 12753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:29.295490 12753 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:02:29.297473 12753 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:29.311693 12753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Generated new TSK 0
I20260619 01:02:29.313176 12753 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:02:30.036448 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:30 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 01:02:30.465548 12783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:30.466044 12783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:30.466250 12783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:30.477938 12783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:30.478117 12783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:30.478319 12783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:02:30.494035 12783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:30.495682 12783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:30.497027 12783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:30.511703 12793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:30.512354 12794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:30.512447 12796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:30.512686 12783 server_base.cc:1034] running on GCE node
I20260619 01:02:30.513278 12783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:30.514263 12783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:30.515483 12783 hybrid_clock.cc:648] HybridClock initialized: now 1781830950515409 us; error 73 us; skew 500 ppm
I20260619 01:02:30.515834 12783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:30 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:30.519052 12783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:02:30.520700 12783 webserver.cc:460] Webserver started at http://127.0.97.1:33715/ using document root <none> and password file <none>
I20260619 01:02:30.521526 12783 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:30.521653 12783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:30.521955 12783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:02:30 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:30.542928 12783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "c19672094cdc4b96b6a0abbd0c1c1001"
format_stamp: "Formatted at 2026-06-19 01:02:30 on dist-test-slave-3rxd"
server_key: "152c2c49be9e3fab4725a85312884f75"
server_key_iv: "dcd31725da8239c5c69a73b779f3b4df"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:30.544097 12783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c19672094cdc4b96b6a0abbd0c1c1001"
format_stamp: "Formatted at 2026-06-19 01:02:30 on dist-test-slave-3rxd"
server_key: "152c2c49be9e3fab4725a85312884f75"
server_key_iv: "dcd31725da8239c5c69a73b779f3b4df"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:30.568835 12783 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.014s sys 0.008s
I20260619 01:02:30.592094 12803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:30.593751 12783 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.001s
I20260619 01:02:30.593945 12783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c19672094cdc4b96b6a0abbd0c1c1001"
format_stamp: "Formatted at 2026-06-19 01:02:30 on dist-test-slave-3rxd"
server_key: "152c2c49be9e3fab4725a85312884f75"
server_key_iv: "dcd31725da8239c5c69a73b779f3b4df"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:30.594190 12783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:30.662254 12783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:30.667877 12783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:30.668273 12783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:30.671363 12783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:30.677327 12783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:30.677465 12783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:30.677647 12783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:30.677712 12783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:02:30 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:30.772830 12783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44613
I20260619 01:02:30.772887 12915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44613 every 8 connection(s)
I20260619 01:02:30.774423 12783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:02:30.779203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 12783
I20260619 01:02:30.779453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:02:30.780501 388 external_mini_cluster.cc:1428] Setting key 3f06066394b415816d0f827938a2655f
Jun 19 01:02:30 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:30.808794 12919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:30.778197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.1:49237)
0619 01:02:30.782360 (+ 4163us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:30.782427 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:30.784510 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:30.785798 (+ 1288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:30.785809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:30.786389 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:30.787217 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:30.787240 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:30.788945 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:30.788950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:30.790927 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:30.790941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:30.792178 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:30.792184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:30.792485 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:30.795744 (+ 3259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:30.795779 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:30.798772 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:30.803544 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:30.803556 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:30.803578 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:30.804243 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:30.804693 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:30.804701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:30.804707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:30.804996 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:30.805770 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:30.805781 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:30.806275 (+ 494us) client_negotiation.cc:769] Sending connection context
0619 01:02:30.806578 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 01:02:30.807227 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3866,"thread_start_us":3523,"threads_started":1}
I20260619 01:02:30.810065 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:30.777396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49237 (local address 127.0.97.62:41135)
0619 01:02:30.785208 (+ 7812us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:30.785214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:30.785232 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:02:30.785286 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:30.785291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:30.785364 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:30.787388 (+ 2024us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:30.787549 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:30.788775 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:30.791109 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:30.791780 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:30.791909 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:30.799146 (+ 7237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:30.799157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:30.799165 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:30.799223 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:30.803351 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:30.804393 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:30.804401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:30.804407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:30.804565 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:30.805151 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:30.805158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:30.805164 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:30.805552 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:30.805700 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:30.809379 (+ 3679us) server_negotiation.cc:299] Negotiation successful
0619 01:02:30.809622 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7688,"thread_start_us":7555,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:02:30.812513 12916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
I20260619 01:02:30.812858 12916 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:30.813741 12916 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:02:30.816293 12703 ts_manager.cc:194] Registered new tserver with Master: c19672094cdc4b96b6a0abbd0c1c1001 (127.0.97.1:44613)
I20260619 01:02:30.818156 12703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:02:30.846010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:02:31.297720 12922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:31.298226 12922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:31.298451 12922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:31.309871 12922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:31.310053 12922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:31.310299 12922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:02:31.324532 12922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:31.326406 12922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:31.327826 12922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:31.352357 12930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:31.352401 12929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:31.362495 12932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:31.363368 12922 server_base.cc:1034] running on GCE node
I20260619 01:02:31.364063 12922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:31.365089 12922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:31.366394 12922 hybrid_clock.cc:648] HybridClock initialized: now 1781830951366299 us; error 77 us; skew 500 ppm
I20260619 01:02:31.366779 12922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:31.370045 12922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:02:31.371977 12922 webserver.cc:460] Webserver started at http://127.0.97.2:41363/ using document root <none> and password file <none>
I20260619 01:02:31.372713 12922 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:31.372850 12922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:31.373174 12922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:31.392395 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "594649c88b9b48319e5015b4f18d02aa"
format_stamp: "Formatted at 2026-06-19 01:02:31 on dist-test-slave-3rxd"
server_key: "a0afb28c8bcac4e332c8ea928817b5b2"
server_key_iv: "ac1f7beef3d83fb8d566b98ab1926286"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:31.393412 12922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "594649c88b9b48319e5015b4f18d02aa"
format_stamp: "Formatted at 2026-06-19 01:02:31 on dist-test-slave-3rxd"
server_key: "a0afb28c8bcac4e332c8ea928817b5b2"
server_key_iv: "ac1f7beef3d83fb8d566b98ab1926286"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:31.409586 12922 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.011s sys 0.003s
I20260619 01:02:31.443042 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:31.444921 12922 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.013s sys 0.004s
I20260619 01:02:31.445188 12922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "594649c88b9b48319e5015b4f18d02aa"
format_stamp: "Formatted at 2026-06-19 01:02:31 on dist-test-slave-3rxd"
server_key: "a0afb28c8bcac4e332c8ea928817b5b2"
server_key_iv: "ac1f7beef3d83fb8d566b98ab1926286"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:31.445452 12922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:31.468021 12922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:31.472114 12922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:31.472405 12922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:31.475112 12922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:31.479408 12922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:31.479521 12922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:31.479671 12922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:31.479740 12922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:31.558362 12922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44697
I20260619 01:02:31.558362 13058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44697 every 8 connection(s)
I20260619 01:02:31.560071 12922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:02:31.568523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 12922
I20260619 01:02:31.568733 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:02:31.569458 388 external_mini_cluster.cc:1428] Setting key 8a8598a6a1e0eec918e2c0b8a23d9f98
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:31.598786 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:31.567304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52909 (local address 127.0.97.62:41135)
0619 01:02:31.575101 (+ 7797us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:31.575108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:31.579520 (+ 4412us) server_negotiation.cc:407] Connection header received
0619 01:02:31.579591 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:31.579599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:31.579699 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:31.579807 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:31.581992 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:31.583261 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:31.585093 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:31.585867 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:31.586002 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:31.590833 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:31.590843 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:31.590852 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:31.590907 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:31.594811 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:31.595808 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:31.595816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:31.595823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:31.595975 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:31.596562 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:31.596567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:31.596571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:31.596920 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:31.597056 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:31.598045 (+ 989us) server_negotiation.cc:299] Negotiation successful
0619 01:02:31.598273 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7662,"thread_start_us":7546,"threads_started":1}
I20260619 01:02:31.600119 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:31.566478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.2:52909)
0619 01:02:31.575303 (+ 8825us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:31.575358 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:31.577999 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:31.580335 (+ 2336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:31.580348 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:31.580976 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:31.581767 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:31.581792 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:31.583456 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:31.583462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:31.584927 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:31.584957 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:31.586270 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:31.586279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:31.586598 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:31.587415 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:31.587449 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:31.590504 (+ 3055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:31.594990 (+ 4486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:31.595002 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:31.595024 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:31.595672 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:31.596124 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:31.596130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:31.596136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:31.596411 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:31.597087 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:31.597097 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:31.597594 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 01:02:31.597929 (+ 335us) client_negotiation.cc:240] Negotiation successful
0619 01:02:31.598580 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8501,"thread_start_us":11948,"threads_started":1}
I20260619 01:02:31.601794 13059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:02:31.602125 13059 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:31.602979 13059 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:31.604619 12703 ts_manager.cc:194] Registered new tserver with Master: 594649c88b9b48319e5015b4f18d02aa (127.0.97.2:44697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260619 01:02:31.605774 12703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:02:31.633247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:31.821712 12916 heartbeater.cc:503] Master 127.0.97.62:41135 was elected leader, sending a full tablet report...
W20260619 01:02:31.953327 13065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:31.953609 13065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:31.953760 13065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:31.960821 13065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:31.960932 13065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:31.961112 13065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:02:31.970467 13065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:31.971676 13065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:31.972991 13065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:31.985692 13070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:31.985666 13071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:31.985646 13073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:31.986517 13065 server_base.cc:1034] running on GCE node
I20260619 01:02:31.987176 13065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:31.988057 13065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:31.989277 13065 hybrid_clock.cc:648] HybridClock initialized: now 1781830951989235 us; error 68 us; skew 500 ppm
I20260619 01:02:31.989585 13065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:31.992843 13065 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:02:31.994282 13065 webserver.cc:460] Webserver started at http://127.0.97.3:41337/ using document root <none> and password file <none>
I20260619 01:02:31.995072 13065 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:31.995220 13065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:31.995566 13065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 01:02:31 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:32.012279 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "da9356ff4b3941a8b817ddfaa4c9654f"
format_stamp: "Formatted at 2026-06-19 01:02:32 on dist-test-slave-3rxd"
server_key: "1be2eeec566f47157fe77b69f0497fc2"
server_key_iv: "e2876d2d908570f072d535ae93c97d63"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:32.013077 13065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da9356ff4b3941a8b817ddfaa4c9654f"
format_stamp: "Formatted at 2026-06-19 01:02:32 on dist-test-slave-3rxd"
server_key: "1be2eeec566f47157fe77b69f0497fc2"
server_key_iv: "e2876d2d908570f072d535ae93c97d63"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:32.018177 13065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:02:32.032330 13080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:32.033877 13065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:02:32.034022 13065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "da9356ff4b3941a8b817ddfaa4c9654f"
format_stamp: "Formatted at 2026-06-19 01:02:32 on dist-test-slave-3rxd"
server_key: "1be2eeec566f47157fe77b69f0497fc2"
server_key_iv: "e2876d2d908570f072d535ae93c97d63"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:32.034222 13065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:32.064831 13065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:32.068521 13065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:32.068758 13065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:32.069676 13065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:32.070951 13065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:32.071023 13065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:32.071115 13065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:32.071158 13065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:32.103468 13065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37673
I20260619 01:02:32.103511 13192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37673 every 8 connection(s)
I20260619 01:02:32.104774 13065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:02:32.114208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13065
I20260619 01:02:32.114377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:02:32.114955 388 external_mini_cluster.cc:1428] Setting key 31c8c4c67c456d3f55cd5143da6355e8
Jun 19 01:02:32 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:32.124779 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:32.107552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35119 (local address 127.0.97.62:41135)
0619 01:02:32.108021 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:32.108026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:32.109054 (+ 1028us) server_negotiation.cc:407] Connection header received
0619 01:02:32.110812 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:32.110818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:32.110910 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:32.111030 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:32.112401 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.113120 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.115181 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.115711 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.115854 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:32.119384 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:32.119389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:32.119393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:32.119433 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:32.122134 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.122827 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.122831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.122834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.122954 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.123355 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.123359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.123360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.123606 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:32.123710 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:32.124370 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 01:02:32.124521 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":216,"threads_started":1}
I20260619 01:02:32.125662 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:32.108158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.3:35119)
0619 01:02:32.108882 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:32.108924 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:32.110514 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:32.111267 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:32.111275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:32.111666 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:32.112230 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:32.112246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.113264 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.113271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:32.114983 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:32.115006 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.115936 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.115942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:32.116251 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:32.116896 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:32.116918 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:32.119202 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:32.122243 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:32.122248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:32.122258 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:32.122698 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:32.123042 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:32.123046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:32.123048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:32.123259 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:32.123708 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:32.123714 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:32.123982 (+ 268us) client_negotiation.cc:769] Sending connection context
0619 01:02:32.124270 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 01:02:32.124622 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":303,"threads_started":1}
I20260619 01:02:32.126905 13193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
I20260619 01:02:32.127193 13193 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:32.127861 13193 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:32.129309 12703 ts_manager.cc:194] Registered new tserver with Master: da9356ff4b3941a8b817ddfaa4c9654f (127.0.97.3:37673)
I20260619 01:02:32.130407 12703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35119
I20260619 01:02:32.130617 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:02:32.132479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 12783
Jun 19 01:02:32 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:02:32.148048 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:32.133462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.0.97.1:44613)
0619 01:02:32.134078 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:32.134083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:32.134105 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:02:32.134157 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:32.134162 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:32.134327 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:32.134440 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:32.135806 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.136756 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.138317 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.138900 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.138995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:32.142950 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:32.142961 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:32.142972 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:32.143007 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:32.145592 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.146202 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.146209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.146211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.146347 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.146666 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.146669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.146671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.147020 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:32.147236 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:32.147467 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 01:02:32.147652 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":322,"threads_started":1}
I20260619 01:02:32.149260 12911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:32.232483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 12783
I20260619 01:02:32.248504 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 12922
Jun 19 01:02:32 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:02:32.264197 13200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:32.249390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.0.97.2:44697)
0619 01:02:32.249937 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:32.249944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:32.249969 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:02:32.250038 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:32.250047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:32.250275 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:32.250472 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:32.251777 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.252781 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.254062 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.254596 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.254696 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:32.258889 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:32.258899 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:32.258913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:32.258945 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:32.261724 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.262318 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.262325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.262327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.262453 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.262787 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.262795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.262800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.263142 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:32.263370 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:32.263596 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 01:02:32.263796 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260619 01:02:32.265426 13054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:32.347644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 12922
I20260619 01:02:32.363199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13065
Jun 19 01:02:32 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:02:32.378674 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:32.364207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.0.97.3:37673)
0619 01:02:32.364829 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:32.364836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:32.364864 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:02:32.364919 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:32.364925 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:32.365142 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:32.365267 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:32.366636 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.367672 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.368923 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:32.369453 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:32.369555 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:32.373843 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:32.373853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:32.373868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:32.373901 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:32.376414 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.376973 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.376979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.376982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.377097 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:32.377396 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:32.377399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:32.377401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:32.377710 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:32.377935 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:32.378107 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 01:02:32.378282 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":322,"threads_started":1}
I20260619 01:02:32.380059 13188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:32.462445 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13065
I20260619 01:02:32.477892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 12671
I20260619 01:02:32.479332 12733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:32.582864 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 12671
I20260619 01:02:32.606395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41135
--webserver_interface=127.0.97.62
--webserver_port=44619
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41135
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:32.918107 13205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:32.918412 13205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:32.918566 13205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:32.925468 13205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:32.925597 13205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:32.925715 13205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:32.925817 13205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:32.925906 13205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:32.935478 13205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41135
--ranger_config_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:32.936633 13205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:32.937889 13205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:32.949182 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:32.949250 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:32.949268 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:32.955220 13205 server_base.cc:1034] running on GCE node
I20260619 01:02:32.955883 13205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:32.957129 13205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:32.958349 13205 hybrid_clock.cc:648] HybridClock initialized: now 1781830952958308 us; error 52 us; skew 500 ppm
I20260619 01:02:32.958676 13205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:32 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:32.961592 13205 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:32.963142 13205 webserver.cc:460] Webserver started at http://127.0.97.62:44619/ using document root <none> and password file <none>
I20260619 01:02:32.963807 13205 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:32.963912 13205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:02:33 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:33.045696 13205 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260619 01:02:33.048643 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:33.049907 13205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:02:33.050086 13205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "a8dce4de2b674690b4254d15c10684f8"
format_stamp: "Formatted at 2026-06-19 01:02:29 on dist-test-slave-3rxd"
server_key: "407a87e1b2d5d17ac930e1cad738a09d"
server_key_iv: "0339c53b8d64f029a50c2ab04309603c"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:33.051120 13205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:33.064481 13205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:33.067902 13205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:33.068167 13205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:33.085453 13205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41135
I20260619 01:02:33.085475 13272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41135 every 8 connection(s)
I20260619 01:02:33.086895 13205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:33.088522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13205
I20260619 01:02:33.089573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44613
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41135
--webserver_port=33715
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:33.096599 13273 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:02:33.100101 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Bootstrap starting.
I20260619 01:02:33.118330 13273 log.cc:824] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:33.126614 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:02:33.127167 13273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Bootstrap complete.
I20260619 01:02:33.135804 13273 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:33.136689 13273 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8dce4de2b674690b4254d15c10684f8, State: Initialized, Role: FOLLOWER
I20260619 01:02:33.137271 13273 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:33.137483 13273 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:33.137589 13273 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:33.137761 13273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:02:33.141294 13273 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:33.141758 13273 leader_election.cc:302] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8dce4de2b674690b4254d15c10684f8; no voters:
I20260619 01:02:33.142349 13273 leader_election.cc:288] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:02:33.142494 13277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:02:33.143548 13277 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [term 2 LEADER]: Becoming Leader. State: Replica: a8dce4de2b674690b4254d15c10684f8, State: Running, Role: LEADER
I20260619 01:02:33.144155 13277 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } }
I20260619 01:02:33.144461 13273 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:33.146863 13278 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a8dce4de2b674690b4254d15c10684f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } } }
I20260619 01:02:33.146878 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8dce4de2b674690b4254d15c10684f8. Latest consensus state: current_term: 2 leader_uuid: "a8dce4de2b674690b4254d15c10684f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8dce4de2b674690b4254d15c10684f8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41135 } } }
I20260619 01:02:33.147214 13278 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:33.147272 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:33.148223 13282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:33.157146 13282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:33.158927 13282 catalog_manager.cc:1259] Loaded cluster ID: 0342432b2ece490c883454898b602d9e
I20260619 01:02:33.159103 13282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:33.164815 13282 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:33.166671 13282 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Loaded TSK: 0
I20260619 01:02:33.168635 13282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:02:33.173471 13273 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 01:02:33.531075 13275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:33.531550 13275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:33.531750 13275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:33.541620 13275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:33.541770 13275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:33.541967 13275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:02:33.553699 13275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33715
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:33.555335 13275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:33.557008 13275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:33.585929 13310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:33.586804 13311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:33.594604 13275 server_base.cc:1034] running on GCE node
W20260619 01:02:33.594931 13314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:33.595767 13275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:33.597105 13275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:33.598723 13275 hybrid_clock.cc:648] HybridClock initialized: now 1781830953598298 us; error 399 us; skew 500 ppm
I20260619 01:02:33.599159 13275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:33 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:33.603395 13275 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:02:33.612562 13275 webserver.cc:460] Webserver started at http://127.0.97.1:33715/ using document root <none> and password file <none>
I20260619 01:02:33.613523 13275 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:33.613718 13275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:02:33 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:33.656908 13275 fs_manager.cc:709] Time spent opening directory manager: real 0.023s user 0.007s sys 0.003s
I20260619 01:02:33.660876 13321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:33.662689 13275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260619 01:02:33.662894 13275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c19672094cdc4b96b6a0abbd0c1c1001"
format_stamp: "Formatted at 2026-06-19 01:02:30 on dist-test-slave-3rxd"
server_key: "152c2c49be9e3fab4725a85312884f75"
server_key_iv: "dcd31725da8239c5c69a73b779f3b4df"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:33.664425 13275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:33.683929 13275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:33.689400 13275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:33.689741 13275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:33.690970 13275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:33.694839 13275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:33.694983 13275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:33.695118 13275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:33.695255 13275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:33.765656 13275 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44613
I20260619 01:02:33.765712 13434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44613 every 8 connection(s)
I20260619 01:02:33.767506 13275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:02:33.770318 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13275
I20260619 01:02:33.771119 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44697
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41135
--webserver_port=41363
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:33 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:33.817687 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:33.777881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.1:54075)
0619 01:02:33.788892 (+ 11011us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:33.788946 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:33.791390 (+ 2444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:33.793712 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:33.793724 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:33.794417 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:33.795301 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:33.795326 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:33.797643 (+ 2317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:33.797651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:33.800994 (+ 3343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:33.801009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:33.802499 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:33.802507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:33.805076 (+ 2569us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:33.806416 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:33.806452 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:33.809174 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:33.812983 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:33.812991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:33.813004 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:33.813400 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:33.813856 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:33.813860 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:33.813864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:33.814099 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:33.815052 (+ 953us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:33.815060 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:33.815438 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 01:02:33.815768 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 01:02:33.816306 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10702,"thread_start_us":10409,"threads_started":1}
I20260619 01:02:33.819887 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:33.779274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54075 (local address 127.0.97.62:41135)
0619 01:02:33.788288 (+ 9014us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:33.788307 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:33.790818 (+ 2511us) server_negotiation.cc:407] Connection header received
0619 01:02:33.792086 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:33.792109 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:33.792583 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:33.792984 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:33.795536 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:33.797448 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:33.801181 (+ 3733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:33.802155 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:33.802342 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:33.809472 (+ 7130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:33.809484 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:33.809503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:33.809556 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:33.812832 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:33.813565 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:33.813572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:33.813577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:33.813727 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:33.814236 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:33.814242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:33.814245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:33.814905 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:33.815100 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:33.818508 (+ 3408us) server_negotiation.cc:299] Negotiation successful
0619 01:02:33.818921 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8673,"thread_start_us":8436,"threads_started":1}
I20260619 01:02:33.822768 13435 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
I20260619 01:02:33.823081 13435 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:33.823913 13435 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
W20260619 01:02:33.827020 13435 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41135 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:02:33.827250 13435 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:33.827513 13435 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:33.828660 13435 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:33.829144 13435 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
W20260619 01:02:33.830605 13435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:02:34.188812 13439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:34.189111 13439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:34.189280 13439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:34.196602 13439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:34.196764 13439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:34.196979 13439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:02:34.209156 13439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=41363
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:34.210805 13439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:34.212116 13439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:34.228935 13453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:34.229544 13454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:34.231057 13456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:34.232223 13439 server_base.cc:1034] running on GCE node
I20260619 01:02:34.232843 13439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:34.236001 13439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:34.237226 13439 hybrid_clock.cc:648] HybridClock initialized: now 1781830954237167 us; error 55 us; skew 500 ppm
I20260619 01:02:34.237536 13439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:34.241726 13439 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:02:34.243541 13439 webserver.cc:460] Webserver started at http://127.0.97.2:41363/ using document root <none> and password file <none>
I20260619 01:02:34.244447 13439 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:34.244593 13439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:34.292879 13439 fs_manager.cc:709] Time spent opening directory manager: real 0.025s user 0.008s sys 0.016s
I20260619 01:02:34.306270 13464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:34.308274 13439 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.010s sys 0.001s
I20260619 01:02:34.308467 13439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "594649c88b9b48319e5015b4f18d02aa"
format_stamp: "Formatted at 2026-06-19 01:02:31 on dist-test-slave-3rxd"
server_key: "a0afb28c8bcac4e332c8ea928817b5b2"
server_key_iv: "ac1f7beef3d83fb8d566b98ab1926286"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:34.309653 13439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:34.328282 13439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:34.333348 13439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:34.333739 13439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:34.336654 13439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:34.338516 13439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:34.338680 13439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:34.338850 13439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:34.338958 13439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:34.427079 13439 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44697
I20260619 01:02:34.427130 13578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44697 every 8 connection(s)
I20260619 01:02:34.429044 13439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:02:34.434693 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13439
I20260619 01:02:34.435463 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:59063/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37673
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41135
--webserver_port=41337
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:34.458446 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:34.433049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55511 (local address 127.0.97.62:41135)
0619 01:02:34.436369 (+ 3320us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:34.436374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:34.436397 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:02:34.438821 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:34.438828 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:34.438908 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:34.439008 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:34.442666 (+ 3658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:34.443898 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:34.446371 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:34.447076 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:34.447170 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:34.451569 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:34.451576 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:34.451581 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:34.451619 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:34.455043 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:34.455898 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:34.455904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:34.455908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:34.456060 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:34.456533 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:34.456538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:34.456542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:34.456869 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:34.456980 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:34.457879 (+ 899us) server_negotiation.cc:299] Negotiation successful
0619 01:02:34.458091 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3200,"thread_start_us":9680,"threads_started":1}
I20260619 01:02:34.461824 13581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:34.432407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.2:55511)
0619 01:02:34.433879 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:34.433934 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:34.438068 (+ 4134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:34.441029 (+ 2961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:34.441040 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:34.441680 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:34.442428 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:34.442456 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:34.444153 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:34.444159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:34.446219 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:34.446233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:34.447543 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:34.447549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:34.447858 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:34.448628 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:34.448657 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:34.451323 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:34.455193 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:34.455201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:34.455219 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:34.455767 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:34.456198 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:34.456202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:34.456206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:34.456411 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:34.457077 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:34.457084 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:34.457456 (+ 372us) client_negotiation.cc:769] Sending connection context
0619 01:02:34.458984 (+ 1528us) client_negotiation.cc:240] Negotiation successful
0619 01:02:34.460536 (+ 1552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":919,"threads_started":1}
I20260619 01:02:34.463903 13579 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
I20260619 01:02:34.464252 13579 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:34.465106 13579 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:34.467164 13238 ts_manager.cc:194] Registered new tserver with Master: 594649c88b9b48319e5015b4f18d02aa (127.0.97.2:44697)
I20260619 01:02:34.469529 13238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55511
I20260619 01:02:34.830772 13435 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:34.831182 13435 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:34.832474 13238 ts_manager.cc:194] Registered new tserver with Master: c19672094cdc4b96b6a0abbd0c1c1001 (127.0.97.1:44613)
I20260619 01:02:34.833546 13238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54075
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260619 01:02:34.889410 13583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:34.889837 13583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:34.890025 13583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:34.900653 13583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:34.900801 13583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:34.901026 13583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:02:34.914942 13583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:59063/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=41337
--tserver_master_addrs=127.0.97.62:41135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:34.916785 13583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:34.918545 13583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:34.943427 13589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:34.943488 13590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:34.955996 13592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:34.963495 13583 server_base.cc:1034] running on GCE node
I20260619 01:02:34.964217 13583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:34.970618 13583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:34.972541 13583 hybrid_clock.cc:648] HybridClock initialized: now 1781830954972435 us; error 106 us; skew 500 ppm
I20260619 01:02:34.972939 13583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:34 dist-test-slave-3rxd krb5kdc[12227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:34.977133 13583 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:02:35.000247 13583 webserver.cc:460] Webserver started at http://127.0.97.3:41337/ using document root <none> and password file <none>
I20260619 01:02:35.001179 13583 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:35.001336 13583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 01:02:35 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:35.023717 13583 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:02:35.026544 13600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:35.027961 13583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:02:35.028316 13583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "da9356ff4b3941a8b817ddfaa4c9654f"
format_stamp: "Formatted at 2026-06-19 01:02:32 on dist-test-slave-3rxd"
server_key: "1be2eeec566f47157fe77b69f0497fc2"
server_key_iv: "e2876d2d908570f072d535ae93c97d63"
server_key_version: "kuduclusterkey@0"
I20260619 01:02:35.029268 13583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:35.043382 13583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:35.047540 13583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:35.047807 13583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:35.048756 13583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:35.050189 13583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:35.050359 13583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 01:02:35.050477 13583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:35.050559 13583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:35.116492 13583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37673
I20260619 01:02:35.116566 13712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37673 every 8 connection(s)
I20260619 01:02:35.118283 13583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:02:35.121670 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13583
Jun 19 01:02:35 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:35.155407 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.122275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42169 (local address 127.0.97.62:41135)
0619 01:02:35.131086 (+ 8811us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.131091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.131112 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:02:35.136373 (+ 5261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.136383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.136523 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.136625 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:35.140568 (+ 3943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.141692 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.143848 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.144469 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.144564 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.148764 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:35.148770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:35.148775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:35.148823 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:35.151986 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.152812 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.152817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.152821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.152966 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.153453 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.153458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.153461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.153792 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:35.153916 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.154812 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.155013 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8700,"thread_start_us":8607,"threads_started":1}
I20260619 01:02:35.157588 13715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.121596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41135 (local address 127.0.97.3:42169)
0619 01:02:35.127273 (+ 5677us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.127329 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.130468 (+ 3139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.137339 (+ 6871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.137356 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.138136 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:35.139174 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.139204 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.141853 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.141858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.143691 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.143705 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.144943 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.144951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.145222 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.146037 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:35.146065 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:35.148515 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:35.152161 (+ 3646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:35.152169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:35.152185 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:35.152682 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:35.153105 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:35.153119 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:35.153122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:35.153333 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:35.154000 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:35.154007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:35.154404 (+ 397us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.155693 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.156217 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5399,"thread_start_us":5160,"threads_started":1}
I20260619 01:02:35.159430 13713 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41135
I20260619 01:02:35.159744 13713 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:35.160604 13713 heartbeater.cc:511] Master 127.0.97.62:41135 requested a full tablet report, sending...
I20260619 01:02:35.162225 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.131076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34282 (local address 127.0.97.62:41135)
0619 01:02:35.141060 (+ 9984us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.141066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.141085 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:02:35.141144 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.141149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.141229 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.141334 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:35.143371 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.144556 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.146462 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.147115 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.148080 (+ 965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.152229 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:35.152236 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:35.152240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:35.152276 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:35.156018 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.156924 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.156930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.156933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.157082 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.157537 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.157543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.157546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.158469 (+ 923us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:35.158765 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.161611 (+ 2846us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.161857 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9785,"thread_start_us":9651,"threads_started":1}
I20260619 01:02:35.162261 13238 ts_manager.cc:194] Registered new tserver with Master: da9356ff4b3941a8b817ddfaa4c9654f (127.0.97.3:37673)
I20260619 01:02:35.163230 13238 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42169
I20260619 01:02:35.166478 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:02:35.210305 13718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.181901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.0.97.62:41135)
0619 01:02:35.182075 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.182080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.191145 (+ 9065us) server_negotiation.cc:407] Connection header received
0619 01:02:35.195024 (+ 3879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.195032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.195111 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.195185 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:35.197228 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.198146 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.199980 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.200605 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.200702 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.203980 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:35.203986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:35.203991 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:35.204031 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:35.207511 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.208247 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.208252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.208256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.208402 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.208778 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.208783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.208786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.209061 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:35.209304 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.209598 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.209808 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 01:02:35 dist-test-slave-3rxd krb5kdc[12227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830954, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260619 01:02:35.215992 13237 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:02:35.230052 13237 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:02:35.338274 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.287775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.0.97.1:44613)
0619 01:02:35.320974 (+ 33199us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.320987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.321037 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:02:35.321158 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.321171 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.321499 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.321736 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:35.326904 (+ 5168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.328386 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.331746 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.333203 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.333318 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.333555 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.337801 (+ 4246us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.337964 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32965,"thread_start_us":34906,"threads_started":1}
I20260619 01:02:35.338645 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.287484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44613 (local address 127.0.0.1:43244)
0619 01:02:35.307313 (+ 19829us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.307337 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.307481 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.322068 (+ 14587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.322076 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.322109 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:35.322544 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.322558 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.328556 (+ 5998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.328561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.331546 (+ 2985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.331561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.334666 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.334672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.336920 (+ 2248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.336989 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.337110 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.337305 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19656,"thread_start_us":15482,"threads_started":1}
I20260619 01:02:35.339514 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.304235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44697 (local address 127.0.0.1:45142)
0619 01:02:35.317383 (+ 13148us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.317406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.317517 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.328556 (+ 11039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.328561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.328588 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:35.329066 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.329079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.334678 (+ 5599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.334682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.336688 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.336704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.338772 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.338781 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.338964 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.338995 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.339088 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.339218 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12996,"thread_start_us":10352,"threads_started":1}
I20260619 01:02:35.344861 13369 tablet_service.cc:1467] Processing CreateTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 (DEFAULT_TABLE table=test-table [id=23f2e24287ad4b79aed347060bf631cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:35.345571 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.287376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37673 (local address 127.0.0.1:47300)
0619 01:02:35.303364 (+ 15988us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.303412 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.303606 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.316372 (+ 12766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.316385 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.316426 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:35.316961 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.316976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.330849 (+ 13873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.330855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.332877 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.332893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.344718 (+ 11825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.344730 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.344908 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.344941 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.345041 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.345177 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15677,"thread_start_us":19572,"threads_started":1}
I20260619 01:02:35.346695 13369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022b4b7c0f51467e970c02e3de3ae244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:35.347410 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.304269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.0.97.2:44697)
0619 01:02:35.326180 (+ 21911us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.326188 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.326216 (+ 28us) server_negotiation.cc:407] Connection header received
0619 01:02:35.326302 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.326310 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.328157 (+ 1847us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.328330 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:35.329263 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.334528 (+ 5265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.336884 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.338266 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.338388 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.338591 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.345845 (+ 7254us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.346022 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20642,"thread_start_us":20511,"threads_started":1}
I20260619 01:02:35.351089 13514 tablet_service.cc:1467] Processing CreateTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 (DEFAULT_TABLE table=test-table [id=23f2e24287ad4b79aed347060bf631cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:35.352360 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.287863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.0.97.3:37673)
0619 01:02:35.315139 (+ 27276us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.315152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.315200 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:02:35.315331 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.315343 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.315684 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.315937 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:35.325380 (+ 9443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.327943 (+ 2563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.333087 (+ 5144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.340454 (+ 7367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.340577 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.340799 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.351833 (+ 11034us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.352020 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26263,"thread_start_us":26073,"threads_started":1}
I20260619 01:02:35.352490 13514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022b4b7c0f51467e970c02e3de3ae244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:35.356845 13648 tablet_service.cc:1467] Processing CreateTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 (DEFAULT_TABLE table=test-table [id=23f2e24287ad4b79aed347060bf631cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:35.358201 13648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 022b4b7c0f51467e970c02e3de3ae244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:35.369094 13735 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Bootstrap starting.
I20260619 01:02:35.373728 13735 tablet_bootstrap.cc:654] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:35.374861 13735 log.cc:824] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:35.386057 13738 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Bootstrap starting.
I20260619 01:02:35.390897 13735 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: No bootstrap required, opened a new log
I20260619 01:02:35.391203 13735 ts_tablet_manager.cc:1397] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.003s
I20260619 01:02:35.391805 13737 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Bootstrap starting.
I20260619 01:02:35.394065 13738 tablet_bootstrap.cc:654] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:35.395422 13737 tablet_bootstrap.cc:654] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:35.395422 13738 log.cc:824] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:35.396850 13737 log.cc:824] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:35.403738 13737 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: No bootstrap required, opened a new log
I20260619 01:02:35.404161 13737 ts_tablet_manager.cc:1397] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260619 01:02:35.412837 13735 raft_consensus.cc:348] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.413265 13735 raft_consensus.cc:374] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:35.413486 13735 raft_consensus.cc:729] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19672094cdc4b96b6a0abbd0c1c1001, State: Initialized, Role: FOLLOWER
I20260619 01:02:35.414341 13738 tablet_bootstrap.cc:492] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: No bootstrap required, opened a new log
I20260619 01:02:35.414680 13738 ts_tablet_manager.cc:1397] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.016s
I20260619 01:02:35.414179 13735 consensus_queue.cc:260] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.425518 13738 raft_consensus.cc:348] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.426477 13435 heartbeater.cc:503] Master 127.0.97.62:41135 was elected leader, sending a full tablet report...
I20260619 01:02:35.427320 13737 raft_consensus.cc:348] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.427723 13737 raft_consensus.cc:374] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:35.427845 13737 raft_consensus.cc:729] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 594649c88b9b48319e5015b4f18d02aa, State: Initialized, Role: FOLLOWER
I20260619 01:02:35.428553 13737 consensus_queue.cc:260] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.430127 13735 ts_tablet_manager.cc:1428] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Time spent starting tablet: real 0.039s user 0.012s sys 0.005s
I20260619 01:02:35.433149 13738 raft_consensus.cc:374] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:35.433341 13738 raft_consensus.cc:729] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9356ff4b3941a8b817ddfaa4c9654f, State: Initialized, Role: FOLLOWER
I20260619 01:02:35.434190 13738 consensus_queue.cc:260] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.435328 13579 heartbeater.cc:503] Master 127.0.97.62:41135 was elected leader, sending a full tablet report...
I20260619 01:02:35.436112 13737 ts_tablet_manager.cc:1428] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Time spent starting tablet: real 0.032s user 0.014s sys 0.000s
I20260619 01:02:35.437353 13713 heartbeater.cc:503] Master 127.0.97.62:41135 was elected leader, sending a full tablet report...
I20260619 01:02:35.439354 13738 ts_tablet_manager.cc:1428] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Time spent starting tablet: real 0.024s user 0.013s sys 0.000s
W20260619 01:02:35.446455 13580 tablet.cc:2369] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:02:35.535099 13436 tablet.cc:2369] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:02:35.635923 13714 tablet.cc:2369] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:35.643445 13741 raft_consensus.cc:482] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:02:35.643682 13741 raft_consensus.cc:504] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.645394 13741 leader_election.cc:288] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da9356ff4b3941a8b817ddfaa4c9654f (127.0.97.3:37673), 594649c88b9b48319e5015b4f18d02aa (127.0.97.2:44697)
I20260619 01:02:35.651917 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.645754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43095 (local address 127.0.97.3:37673)
0619 01:02:35.645946 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.645955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.646734 (+ 779us) server_negotiation.cc:407] Connection header received
0619 01:02:35.646958 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.646962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.647017 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.647128 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:35.647750 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.648423 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.650090 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.650973 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.651103 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.651219 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.651539 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.651664 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:02:35.651940 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.645818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35843 (local address 127.0.97.2:44697)
0619 01:02:35.646075 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.646083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.646650 (+ 567us) server_negotiation.cc:407] Connection header received
0619 01:02:35.646907 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.646911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.646981 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.647093 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:35.647684 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.648375 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.650062 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.650964 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.651098 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.651223 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.651559 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.651690 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:02:35.652086 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.645703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44697 (local address 127.0.97.1:35843)
0619 01:02:35.646536 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.646552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.646721 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.647106 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.647120 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.647145 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:35.647539 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.647550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.648495 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.648503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.649913 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.649924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.651236 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.651245 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.651410 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.651432 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.651535 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.651653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":22,"thread_start_us":477,"threads_started":1}
I20260619 01:02:35.652217 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.645764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37673 (local address 127.0.97.1:43095)
0619 01:02:35.646637 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:35.646650 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:35.646807 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:35.647139 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:35.647143 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:35.647165 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:35.647574 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.647582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.648555 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.648560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.649938 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:35.649949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.651205 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.651215 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:35.651352 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.651410 (+ 58us) client_negotiation.cc:769] Sending connection context
0619 01:02:35.651514 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:02:35.651634 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":18,"thread_start_us":508,"threads_started":1}
I20260619 01:02:35.653182 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022b4b7c0f51467e970c02e3de3ae244" candidate_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" is_pre_election: true
I20260619 01:02:35.653182 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022b4b7c0f51467e970c02e3de3ae244" candidate_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594649c88b9b48319e5015b4f18d02aa" is_pre_election: true
I20260619 01:02:35.654134 13668 raft_consensus.cc:2454] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19672094cdc4b96b6a0abbd0c1c1001 in term 0.
I20260619 01:02:35.654125 13534 raft_consensus.cc:2454] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c19672094cdc4b96b6a0abbd0c1c1001 in term 0.
I20260619 01:02:35.654867 13325 leader_election.cc:302] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c19672094cdc4b96b6a0abbd0c1c1001, da9356ff4b3941a8b817ddfaa4c9654f; no voters:
I20260619 01:02:35.655493 13741 raft_consensus.cc:2781] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:02:35.655645 13741 raft_consensus.cc:482] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:02:35.655725 13741 raft_consensus.cc:3037] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:35.657759 13741 raft_consensus.cc:504] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.658375 13741 leader_election.cc:288] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [CANDIDATE]: Term 1 election: Requested vote from peers da9356ff4b3941a8b817ddfaa4c9654f (127.0.97.3:37673), 594649c88b9b48319e5015b4f18d02aa (127.0.97.2:44697)
I20260619 01:02:35.658777 13668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022b4b7c0f51467e970c02e3de3ae244" candidate_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9356ff4b3941a8b817ddfaa4c9654f"
I20260619 01:02:35.658890 13534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "022b4b7c0f51467e970c02e3de3ae244" candidate_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "594649c88b9b48319e5015b4f18d02aa"
I20260619 01:02:35.658987 13668 raft_consensus.cc:3037] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:35.659075 13534 raft_consensus.cc:3037] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:35.661990 13534 raft_consensus.cc:2454] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19672094cdc4b96b6a0abbd0c1c1001 in term 1.
I20260619 01:02:35.661993 13668 raft_consensus.cc:2454] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c19672094cdc4b96b6a0abbd0c1c1001 in term 1.
I20260619 01:02:35.662364 13322 leader_election.cc:302] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 594649c88b9b48319e5015b4f18d02aa, c19672094cdc4b96b6a0abbd0c1c1001; no voters:
I20260619 01:02:35.662664 13741 raft_consensus.cc:2781] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:35.663194 13741 raft_consensus.cc:686] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 1 LEADER]: Becoming Leader. State: Replica: c19672094cdc4b96b6a0abbd0c1c1001, State: Running, Role: LEADER
I20260619 01:02:35.663852 13741 consensus_queue.cc:237] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } }
I20260619 01:02:35.668215 13238 catalog_manager.cc:5495] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 reported cstate change: term changed from 0 to 1, leader changed from <none> to c19672094cdc4b96b6a0abbd0c1c1001 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c19672094cdc4b96b6a0abbd0c1c1001" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 } health_report { overall_health: UNKNOWN } } }
I20260619 01:02:35.761404 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.756177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43252 (local address 127.0.97.1:44613)
0619 01:02:35.756415 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.756421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.756708 (+ 287us) server_negotiation.cc:407] Connection header received
0619 01:02:35.756872 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.756876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.756960 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.757078 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:02:35.757409 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.758146 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.759464 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.759847 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.759938 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.760356 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:02:35.760854 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:02:35.760962 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.761029 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.761091 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:02:35.770673 13534 raft_consensus.cc:1261] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 1 FOLLOWER]: Refusing update from remote peer c19672094cdc4b96b6a0abbd0c1c1001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:35.770702 13668 raft_consensus.cc:1261] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 1 FOLLOWER]: Refusing update from remote peer c19672094cdc4b96b6a0abbd0c1c1001: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:35.771727 13741 consensus_queue.cc:1035] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da9356ff4b3941a8b817ddfaa4c9654f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:35.772105 13753 consensus_queue.cc:1035] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [LEADER]: Connected to new peer: Peer: permanent_uuid: "594649c88b9b48319e5015b4f18d02aa" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:35.784502 13757 mvcc.cc:205] Tried to move back new op lower bound from 7298379594826084352 to 7298379594403127296. Current Snapshot: MvccSnapshot[applied={T|T < 7298379594826084352}]
I20260619 01:02:35.797736 13238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:35.798048 13238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:35.801246 13238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8dce4de2b674690b4254d15c10684f8: Sending DeleteTablet for 3 replicas of tablet 022b4b7c0f51467e970c02e3de3ae244
I20260619 01:02:35.802359 13648 tablet_service.cc:1514] Processing DeleteTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47300
I20260619 01:02:35.802459 13514 tablet_service.cc:1514] Processing DeleteTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45142
I20260619 01:02:35.802977 13369 tablet_service.cc:1514] Processing DeleteTablet for tablet 022b4b7c0f51467e970c02e3de3ae244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:35 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43244
I20260619 01:02:35.803941 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13275
I20260619 01:02:35.812547 13767 tablet_replica.cc:330] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: stopping tablet replica
I20260619 01:02:35.813078 13767 raft_consensus.cc:2229] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:02:35.813872 13767 raft_consensus.cc:2258] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:35.817772 13766 tablet_replica.cc:330] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: stopping tablet replica
I20260619 01:02:35.818239 13766 raft_consensus.cc:2229] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:35.818809 13766 raft_consensus.cc:2258] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:35.821201 13768 tablet_replica.cc:330] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: stopping tablet replica
I20260619 01:02:35.821656 13768 raft_consensus.cc:2229] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:35.822156 13768 raft_consensus.cc:2258] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:35.823498 13767 ts_tablet_manager.cc:1905] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:35.823900 13766 ts_tablet_manager.cc:1905] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:35.826782 13768 ts_tablet_manager.cc:1905] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:35.831391 13767 ts_tablet_manager.cc:1918] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:35.831605 13767 log.cc:1196] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/022b4b7c0f51467e970c02e3de3ae244
I20260619 01:02:35.832177 13767 ts_tablet_manager.cc:1939] T 022b4b7c0f51467e970c02e3de3ae244 P c19672094cdc4b96b6a0abbd0c1c1001: Deleting consensus metadata
I20260619 01:02:35.832374 13766 ts_tablet_manager.cc:1918] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:35.832602 13766 log.cc:1196] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/022b4b7c0f51467e970c02e3de3ae244
I20260619 01:02:35.833256 13766 ts_tablet_manager.cc:1939] T 022b4b7c0f51467e970c02e3de3ae244 P da9356ff4b3941a8b817ddfaa4c9654f: Deleting consensus metadata
I20260619 01:02:35.833987 13222 catalog_manager.cc:4841] TS c19672094cdc4b96b6a0abbd0c1c1001 (127.0.97.1:44613): tablet 022b4b7c0f51467e970c02e3de3ae244 (table test-table [id=23f2e24287ad4b79aed347060bf631cc]) successfully deleted
I20260619 01:02:35.835677 13225 catalog_manager.cc:4841] TS da9356ff4b3941a8b817ddfaa4c9654f (127.0.97.3:37673): tablet 022b4b7c0f51467e970c02e3de3ae244 (table test-table [id=23f2e24287ad4b79aed347060bf631cc]) successfully deleted
I20260619 01:02:35.836903 13768 ts_tablet_manager.cc:1918] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:35.837067 13768 log.cc:1196] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/022b4b7c0f51467e970c02e3de3ae244
I20260619 01:02:35.837568 13768 ts_tablet_manager.cc:1939] T 022b4b7c0f51467e970c02e3de3ae244 P 594649c88b9b48319e5015b4f18d02aa: Deleting consensus metadata
I20260619 01:02:35.839186 13222 catalog_manager.cc:4841] TS 594649c88b9b48319e5015b4f18d02aa (127.0.97.2:44697): tablet 022b4b7c0f51467e970c02e3de3ae244 (table test-table [id=23f2e24287ad4b79aed347060bf631cc]) successfully deleted
I20260619 01:02:35.841071 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.805159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.0.97.1:44613)
0619 01:02:35.805384 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.805392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.818274 (+ 12882us) server_negotiation.cc:407] Connection header received
0619 01:02:35.818471 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.818477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.818578 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.818699 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:35.820837 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.822096 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.824005 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.824721 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.824851 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.833552 (+ 8701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:35.833566 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:35.833586 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:35.833630 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:35.837880 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.838791 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.838800 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.838804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.838993 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.839423 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.839428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.839432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.839889 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:35.840167 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.840458 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.840696 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:02:35.841799 13430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:35.937733 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13275
I20260619 01:02:35.955972 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13439
I20260619 01:02:35.968333 13734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:35.956711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.0.97.2:44697)
0619 01:02:35.956867 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:35.956872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:35.956891 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:02:35.957071 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:35.957075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:35.957137 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:35.957227 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:35.958472 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.959201 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.960423 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:35.960783 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:35.960868 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:35.963339 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:35.963351 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:35.963367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:35.963397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:35.965941 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.966533 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.966539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.966542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.966666 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:35.967067 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:35.967070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:35.967072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:35.967394 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:35.967613 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:35.967916 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 01:02:35.968073 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:02:35.968986 13574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:36.057265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13439
I20260619 01:02:36.073717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13583
I20260619 01:02:36.087146 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:36.074491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.0.97.3:37673)
0619 01:02:36.074679 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:36.074687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:36.074730 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:02:36.074968 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:36.074972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:36.075040 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:36.075151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:36.076398 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:36.077108 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:36.078437 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:36.079001 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:36.079166 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:36.081601 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:36.081615 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:36.081631 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:36.081663 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:36.084400 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:36.085061 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:36.085068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:36.085070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:36.085199 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:36.085588 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:36.085593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:36.085598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:36.085944 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:36.086246 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:36.086546 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 01:02:36.086782 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:02:36.087812 13708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:36.183084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13583
I20260619 01:02:36.199935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13205
I20260619 01:02:36.201085 13268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:36.310891 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13205
I20260619 01:02:36.334396 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 01:02:36.669584 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 01:02:36.669763 388 mini_ranger.cc:66] Stopping Ranger...
I20260619 01:02:37.014933 388 mini_ranger.cc:68] Stopped Ranger
2026-06-19 01:02:37.015 UTC [12257] LOG: received smart shutdown request
2026-06-19 01:02:37.018 UTC [12257] LOG: background worker "logical replication launcher" (PID 12266) exited with exit code 1
2026-06-19 01:02:37.018 UTC [12261] LOG: shutting down
2026-06-19 01:02:37.157 UTC [12257] LOG: database system is shut down
2026-06-19T01:02:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50596 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:02:37 dist-test-slave-3rxd krb5kdc[13782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:02:37 dist-test-slave-3rxd krb5kdc[13782](info): set up 2 sockets
Jun 19 01:02:37 dist-test-slave-3rxd krb5kdc[13782](info): commencing operation
krb5kdc: starting...
W20260619 01:02:38.363024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.013s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:02:38 dist-test-slave-3rxd krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:02:38.567836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33893
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:38.896019 13795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:38.896286 13795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:38.896370 13795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:38.903510 13795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:38.903612 13795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:38.903717 13795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:38.903821 13795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:38.903882 13795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:38.913494 13795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:38.914773 13795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:38.916043 13795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:38.927767 13803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:38.927959 13800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:38.927938 13801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:38.928479 13795 server_base.cc:1034] running on GCE node
I20260619 01:02:38.929174 13795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:38.930430 13795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:38.931627 13795 hybrid_clock.cc:648] HybridClock initialized: now 1781830958931552 us; error 64 us; skew 500 ppm
I20260619 01:02:38.931907 13795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:38 dist-test-slave-3rxd krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:38.940081 13795 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:38.941406 13795 webserver.cc:460] Webserver started at http://127.0.97.62:39901/ using document root <none> and password file <none>
I20260619 01:02:38.942065 13795 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:38.942194 13795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:38.942512 13795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:38.945443 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "ccb3e17933f24780a2cb18ae91906f3c"
format_stamp: "Formatted at 2026-06-19 01:02:38 on dist-test-slave-3rxd"
server_key: "87c825e67ece66a2ccd36d4b20df3471"
server_key_iv: "ad71fe071b7aba08bd3641e3b1f0c29a"
server_key_version: "encryptionkey@0"
I20260619 01:02:38.946208 13795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "ccb3e17933f24780a2cb18ae91906f3c"
format_stamp: "Formatted at 2026-06-19 01:02:38 on dist-test-slave-3rxd"
server_key: "87c825e67ece66a2ccd36d4b20df3471"
server_key_iv: "ad71fe071b7aba08bd3641e3b1f0c29a"
server_key_version: "encryptionkey@0"
I20260619 01:02:38.951445 13795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 01:02:38.954535 13810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:38.955827 13795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:02:38.955960 13795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "ccb3e17933f24780a2cb18ae91906f3c"
format_stamp: "Formatted at 2026-06-19 01:02:38 on dist-test-slave-3rxd"
server_key: "87c825e67ece66a2ccd36d4b20df3471"
server_key_iv: "ad71fe071b7aba08bd3641e3b1f0c29a"
server_key_version: "encryptionkey@0"
I20260619 01:02:38.956125 13795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:38.972167 13795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:38.976181 13795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:38.976392 13795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:38.994494 13795 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33893
I20260619 01:02:38.994526 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33893 every 8 connection(s)
I20260619 01:02:38.995849 13795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:38.999377 13862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:39.006085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13795
I20260619 01:02:39.005532 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:02:39.006335 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:02:39.007005 388 external_mini_cluster.cc:1428] Setting key ade20fcc54e44c88e6f947610af51e5b
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
I20260619 01:02:39.026865 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.008751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.0.97.62:33893)
0619 01:02:39.009625 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.009645 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.010047 (+ 402us) server_negotiation.cc:407] Connection header received
0619 01:02:39.012035 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.012040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.012252 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.012447 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.013705 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.014933 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.016265 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.016808 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.016999 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.020161 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.020168 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.020177 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.020202 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.022883 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.023500 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.023508 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.023511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.023667 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.024024 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.024032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.024035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.024451 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.024678 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.025151 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.025482 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":616,"thread_start_us":417,"threads_started":1}
I20260619 01:02:39.026883 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.008753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33893 (local address 127.0.0.1:52728)
0619 01:02:39.009883 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.009919 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.011678 (+ 1759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.012463 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.012470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.013175 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.013498 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.013514 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.015088 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.015095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.016130 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.016140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.017025 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.017032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.017276 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.017908 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:39.017930 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:39.019950 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:39.022996 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.023008 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.023017 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.023379 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.023771 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.023778 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.023782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.023927 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.024612 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:39.024619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:39.024815 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 01:02:39.025046 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 01:02:39.025600 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":182,"thread_start_us":738,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 01:02:39.029881 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:02:39.031122 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.030628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56618)
0619 01:02:39.030841 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.030969 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 01:02:39.031399 13812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
I20260619 01:02:39.033010 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.032556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56630)
0619 01:02:39.032751 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.032839 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:02:39.033226 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34839: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:02:39.052028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34839
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:39.082589 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } attempt: 1
I20260619 01:02:39.083811 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.083372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56636)
0619 01:02:39.083585 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.083688 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 01:02:39.084894 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.084488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56646)
0619 01:02:39.084661 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.084747 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 01:02:39.085115 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34839: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
I20260619 01:02:39.153496 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } attempt: 2
I20260619 01:02:39.154796 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.154171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56654)
0619 01:02:39.154452 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.154588 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 01:02:39.155993 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.155528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56666)
0619 01:02:39.155701 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.155808 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 01:02:39.156252 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34839: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
I20260619 01:02:39.247501 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } attempt: 3
I20260619 01:02:39.248739 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.248167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56680)
0619 01:02:39.248417 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.248538 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 01:02:39.249816 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.249412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56686)
0619 01:02:39.249564 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.249666 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:02:39.250003 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34839: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
W20260619 01:02:39.366050 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:39.366351 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:39.366542 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:39.373931 13869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:39.374096 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:39.374223 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:39.374320 13869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:39.374396 13869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:39.384485 13869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:39.385782 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:39.387185 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:02:39.393234 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } attempt: 4
I20260619 01:02:39.394356 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.393845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56688)
0619 01:02:39.394132 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.394232 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 01:02:39.395351 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.395015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56704)
0619 01:02:39.395163 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.395237 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:02:39.395531 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34839: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34839: connect: Connection refused (error 111)
W20260619 01:02:39.398741 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:39.398813 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:39.398764 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:39.404011 13869 server_base.cc:1034] running on GCE node
I20260619 01:02:39.404847 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:39.406306 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:39.407524 13869 hybrid_clock.cc:648] HybridClock initialized: now 1781830959407475 us; error 60 us; skew 500 ppm
I20260619 01:02:39.407864 13869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:39.411020 13869 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:02:39.412415 13869 webserver.cc:460] Webserver started at http://127.0.97.61:35575/ using document root <none> and password file <none>
I20260619 01:02:39.413125 13869 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:39.413265 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:39.413563 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:39.416504 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data/instance:
uuid: "782f40f3d53e4e36854d377d22118e1a"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "01f50db68d8bf28633791723e9400ef3"
server_key_iv: "88d8a9817ab1991695e909a442174729"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.417273 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal/instance:
uuid: "782f40f3d53e4e36854d377d22118e1a"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "01f50db68d8bf28633791723e9400ef3"
server_key_iv: "88d8a9817ab1991695e909a442174729"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.422152 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:02:39.425443 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:39.426694 13869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:02:39.426909 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "782f40f3d53e4e36854d377d22118e1a"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "01f50db68d8bf28633791723e9400ef3"
server_key_iv: "88d8a9817ab1991695e909a442174729"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.427107 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:39.449968 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:39.454202 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:39.454483 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:39.471887 13869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34839
I20260619 01:02:39.471930 13935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34839 every 8 connection(s)
I20260619 01:02:39.473136 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:02:39.476555 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:39.481511 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13869
I20260619 01:02:39.481724 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-1/wal/instance
I20260619 01:02:39.482105 388 external_mini_cluster.cc:1428] Setting key 2bdf279ca7a1d8ac19533d09c36a24d9
I20260619 01:02:39.482512 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:39.502465 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.484982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.0.97.62:33893)
0619 01:02:39.485148 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.485153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.486510 (+ 1357us) server_negotiation.cc:407] Connection header received
0619 01:02:39.488435 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.488440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.488532 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.488651 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.490027 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.490784 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.492395 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.492809 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.492940 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.496293 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.496298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.496302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.496332 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.499292 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.500013 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.500020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.500025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.500198 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.500593 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.500601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.500606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.500974 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.501145 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.501797 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.502041 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 01:02:39.503129 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.485562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33893 (local address 127.0.0.1:52732)
0619 01:02:39.486340 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.486379 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.488171 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.488872 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.488879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.489356 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.489865 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.489880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.490939 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.490943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.492243 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.492255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.493033 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.493040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.493260 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.493932 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:39.493954 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:39.496077 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:39.499431 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.499437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.499449 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.499864 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.500312 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.500316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.500318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.500479 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.501095 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:39.501100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:39.501404 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 01:02:39.501657 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 01:02:39.502017 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":377,"threads_started":1}
I20260619 01:02:39.504801 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:02:39.523072 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.505583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56710)
0619 01:02:39.505764 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.505779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.505874 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.506984 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.506988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.507415 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.507535 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.507543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.509692 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.509700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.511302 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.511319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.512977 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.512982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.513261 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.513941 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:39.513963 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:39.516357 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:39.520573 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.520577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.520579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.520905 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.521317 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.521320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.521322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.521453 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.522207 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:39.522212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:39.522392 (+ 180us) client_negotiation.cc:769] Sending connection context
0619 01:02:39.522532 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 01:02:39.522819 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 01:02:39.523510 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.505901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.97.61:34839)
0619 01:02:39.506514 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.506520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.506544 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:02:39.506600 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.506606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.506846 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.506996 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.507690 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.509498 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.511477 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.512821 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.512950 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.516579 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.516595 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.516613 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.516659 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.520426 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.521038 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.521058 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.521064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.521212 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.521563 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.521571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.521577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.522042 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.522278 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.522745 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.523022 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":362,"threads_started":1}
I20260619 01:02:39.525897 13936 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:02:39.527020 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.526554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50916)
0619 01:02:39.526769 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.526896 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 01:02:39.527256 13886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.528770 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.528441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50922)
0619 01:02:39.528602 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.528682 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 01:02:39.529011 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 01:02:39.535848 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41909
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:39.578336 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 1
I20260619 01:02:39.579725 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.579138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50938)
0619 01:02:39.579387 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.579512 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 01:02:39.580927 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.580443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50940)
0619 01:02:39.580657 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.580760 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 01:02:39.581157 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.649421 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 2
I20260619 01:02:39.650749 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.650087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50952)
0619 01:02:39.650346 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.650477 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 01:02:39.651948 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.651470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50956)
0619 01:02:39.651663 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.651770 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 01:02:39.652194 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.694797 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } attempt: 5
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:02:39.709995 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.695699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.61:34839)
0619 01:02:39.695852 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.695856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.695877 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:02:39.696075 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.696079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.696156 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.696240 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.697302 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.698039 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.699942 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.700410 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.700513 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.704778 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.704784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.704787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.704820 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.707538 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.708294 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.708298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.708300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.708410 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.708802 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.708805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.708807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.709070 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.709175 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.709602 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.709759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 01:02:39.710147 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.695534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56716)
0619 01:02:39.695771 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.695794 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.695936 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.696313 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.696318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.696881 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.697113 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.697126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.698198 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.698206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.699773 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.699800 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.700604 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.700613 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.700871 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.701652 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:39.701682 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:39.704564 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:39.707672 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.707677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.707681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.708177 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.708516 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.708521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.708525 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.708702 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.709185 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:39.709190 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:39.709362 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 01:02:39.709525 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 01:02:39.709807 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 01:02:39.711298 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:02:39.712330 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.711905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50958)
0619 01:02:39.712085 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.712174 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 01:02:39.713320 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.712885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50970)
0619 01:02:39.713066 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.713178 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 01:02:39.713564 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.743434 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 3
I20260619 01:02:39.744663 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.744112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50986)
0619 01:02:39.744340 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.744453 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 01:02:39.745782 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.745309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50990)
0619 01:02:39.745530 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.745633 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 01:02:39.746016 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.764808 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 1
I20260619 01:02:39.765988 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.765423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:50992)
0619 01:02:39.765660 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.765781 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 01:02:39.767071 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.766649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51004)
0619 01:02:39.766836 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.766927 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:02:39.767321 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.835592 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 2
I20260619 01:02:39.836834 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.836267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51012)
0619 01:02:39.836516 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.836639 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 01:02:39.837846 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.837473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51018)
0619 01:02:39.837635 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.837718 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:02:39.838088 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
W20260619 01:02:39.864024 13943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:39.864389 13943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:39.864586 13943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:39.871841 13943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:39.871989 13943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:39.872087 13943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:39.872189 13943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:39.872274 13943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:39.882249 13943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33893,127.0.97.61:34839,127.0.97.60:41909
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:39.883708 13943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:39.885155 13943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:02:39.889257 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 4
I20260619 01:02:39.890550 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.889948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51020)
0619 01:02:39.890239 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.890345 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 01:02:39.891678 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.891319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51028)
0619 01:02:39.891500 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.891575 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 01:02:39.891954 13936 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
W20260619 01:02:39.898118 13948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:39.898190 13951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:39.898162 13949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:39.898933 13943 server_base.cc:1034] running on GCE node
I20260619 01:02:39.899778 13943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:39.901213 13943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:39.902408 13943 hybrid_clock.cc:648] HybridClock initialized: now 1781830959902372 us; error 26 us; skew 500 ppm
I20260619 01:02:39.902843 13943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:39.905810 13943 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:02:39.907323 13943 webserver.cc:460] Webserver started at http://127.0.97.60:44325/ using document root <none> and password file <none>
I20260619 01:02:39.908001 13943 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:39.908143 13943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:39.908444 13943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:39.911399 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data/instance:
uuid: "0d42278ca3e843e29282266cc9da0580"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "eed5a1e0981fb38f2ccf030a39acf7b0"
server_key_iv: "8f7054e0fdc557d7f28abf48e1f9b77b"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.912159 13943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal/instance:
uuid: "0d42278ca3e843e29282266cc9da0580"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "eed5a1e0981fb38f2ccf030a39acf7b0"
server_key_iv: "8f7054e0fdc557d7f28abf48e1f9b77b"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.917003 13943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:02:39.920233 13958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:39.921422 13943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:02:39.921608 13943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "0d42278ca3e843e29282266cc9da0580"
format_stamp: "Formatted at 2026-06-19 01:02:39 on dist-test-slave-3rxd"
server_key: "eed5a1e0981fb38f2ccf030a39acf7b0"
server_key_iv: "8f7054e0fdc557d7f28abf48e1f9b77b"
server_key_version: "encryptionkey@0"
I20260619 01:02:39.921803 13943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:39.939380 13943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:39.943076 13943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:39.943336 13943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:39.944321 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 3
I20260619 01:02:39.945365 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.944882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51040)
0619 01:02:39.945114 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.945192 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 01:02:39.946300 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.945966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51054)
0619 01:02:39.946145 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.946207 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:02:39.946486 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41909: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41909: connect: Connection refused (error 111)
I20260619 01:02:39.961694 13943 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41909
I20260619 01:02:39.961740 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41909 every 8 connection(s)
I20260619 01:02:39.963066 13943 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:02:39.966037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 13943
I20260619 01:02:39.966239 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781830760188020-388-0/minicluster-data/master-2/wal/instance
I20260619 01:02:39.966807 388 external_mini_cluster.cc:1428] Setting key c4ff8bcab23599a506e529201386dd9a
I20260619 01:02:39.967038 14010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:39.975610 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:02:39 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:39.990252 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.968724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.0.97.60:41909)
0619 01:02:39.969471 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.969483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.969523 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:02:39.971966 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.971988 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.972399 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.972823 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.974301 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.975644 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.977096 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.977852 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.978053 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.983104 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.983112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.983122 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.983159 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.986205 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.987026 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.987041 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.987047 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.987196 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.987604 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.987614 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.987617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.988204 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.988502 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.988916 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.989292 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":350,"threads_started":1}
I20260619 01:02:39.993834 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.977912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33893 (local address 127.0.0.1:52736)
0619 01:02:39.979229 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.979276 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.979432 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.979832 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.979841 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.980682 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.981057 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.981071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.982387 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.982395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.983509 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.983521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.984261 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.984267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:39.984443 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.985075 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:39.985092 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:39.987167 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:39.990095 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.990105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.990111 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.991611 (+ 1500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.992005 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:39.992010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:39.992014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:39.992151 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:39.992719 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:39.992727 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:39.992909 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 01:02:39.993102 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 01:02:39.993429 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":870,"threads_started":1}
I20260619 01:02:39.995419 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.977981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.0.97.62:33893)
0619 01:02:39.978234 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.978242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.979390 (+ 1148us) server_negotiation.cc:407] Connection header received
0619 01:02:39.979549 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.979569 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.979632 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.979774 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.981244 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.982241 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.983654 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.984071 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.984213 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:39.987375 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:39.987380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:39.987383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:39.987413 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:39.989957 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.991756 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.991761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.991765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.991893 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:39.992266 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:39.992271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:39.992274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:39.992596 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:39.992757 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:39.993238 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 01:02:39.993502 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:02:39.996980 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:02:40.012508 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.997708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34839 (local address 127.0.0.1:56722)
0619 01:02:39.997922 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:39.997937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:39.998070 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:39.998527 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:39.998531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:39.998996 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:39.999174 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:39.999185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.000367 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.000371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.001595 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.001609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.002932 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.002941 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.003188 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.003706 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:40.003732 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:40.005982 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:40.009736 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.009741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.009745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.010284 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.010707 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.010732 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.010736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.010925 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.011514 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:40.011518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:40.011687 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 01:02:40.011837 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 01:02:40.012140 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 01:02:40.012548 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.997712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56722 (local address 127.0.97.61:34839)
0619 01:02:39.997875 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.997880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.998015 (+ 135us) server_negotiation.cc:407] Connection header received
0619 01:02:39.998192 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.998198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.998275 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.998370 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.999350 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.000215 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.001766 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.002465 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.002601 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.006176 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.006183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.006188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.006231 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.009615 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.010405 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.010410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.010413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.010576 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.011038 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.011042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.011045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.011382 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.011541 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.011971 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.012187 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:02:40.012599 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:39.993153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.0.97.62:33893)
0619 01:02:39.993732 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:39.993739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:39.995035 (+ 1296us) server_negotiation.cc:407] Connection header received
0619 01:02:39.995181 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:39.995185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:39.995243 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:39.995356 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:39.996924 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:39.998132 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:39.999884 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.000576 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.000696 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.005771 (+ 5075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.005776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.005780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.005809 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.009558 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.010414 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.010419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.010423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.010587 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.011045 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.011050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.011053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.011400 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.011647 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.011965 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.012191 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":231,"threads_started":1}
I20260619 01:02:40.013489 14010 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:02:40.027576 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.014205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.0.97.60:41909)
0619 01:02:40.014343 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.014349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.014421 (+ 72us) server_negotiation.cc:407] Connection header received
0619 01:02:40.014582 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.014588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.014665 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.014796 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.015770 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.016518 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.017777 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.018161 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.018245 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.021671 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.021676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.021679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.021704 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.024829 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.025655 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.025661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.025664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.025807 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.026155 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.026160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.026164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.026542 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.026691 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.027097 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.027348 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 01:02:40.027652 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.014161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51078)
0619 01:02:40.014339 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:40.014357 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:40.014462 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:40.014938 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:40.014941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:40.015447 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:40.015627 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.015639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.016654 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.016658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.017643 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.017657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.018309 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.018315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.018514 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.019178 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:40.019193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:40.021465 (+ 2272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:40.024964 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.024968 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.024970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.025529 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.025925 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.025928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.025931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.026050 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.026675 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:40.026681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:40.026842 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 01:02:40.026962 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 01:02:40.027179 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 01:02:40.029930 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.014681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.0.97.61:34839)
0619 01:02:40.014850 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.014854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.014868 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:02:40.014902 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.014905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.014952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.015021 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.016390 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.017156 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.018614 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.019238 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.019329 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.023685 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.023691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.023695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.023734 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.027134 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.027932 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.027937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.027941 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.028135 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.028482 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.028488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.028493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.028820 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.029119 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.029373 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.029586 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:02:40.033200 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: Bootstrap starting.
I20260619 01:02:40.034734 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:02:40.036573 14010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:40.037720 14010 log.cc:824] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:40.041024 14010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: No bootstrap required, opened a new log
I20260619 01:02:40.056725 14010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.057171 14010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:40.057279 14010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d42278ca3e843e29282266cc9da0580, State: Initialized, Role: FOLLOWER
I20260619 01:02:40.058087 14010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.059504 14015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.039480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.0.97.62:33893)
0619 01:02:40.039703 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.039711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.040333 (+ 622us) server_negotiation.cc:407] Connection header received
0619 01:02:40.040695 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.040704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.040807 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.040882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.043795 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.045185 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.047563 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.048254 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.048360 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.052277 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.052284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.052289 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.052328 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.056023 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.056981 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.056988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.056992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.057142 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.057634 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.057639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.057642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.057984 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.058203 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.058909 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.059143 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:02:40.063362 14024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.063797 14024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.064807 13940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.040039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.97.61:34839)
0619 01:02:40.040219 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.040224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.046740 (+ 6516us) server_negotiation.cc:407] Connection header received
0619 01:02:40.047175 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.047184 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.047263 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.047334 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.049700 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.050871 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.053227 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.053893 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.053997 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.057705 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.057714 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.057722 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.057761 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.061437 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.062148 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.062156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.062160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.062323 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.062990 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.062995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.062998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.063407 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.063624 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.064195 (+ 571us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.064432 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:02:40.064805 14010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:40.072041 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.041040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.0.97.60:41909)
0619 01:02:40.041353 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.041359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.052510 (+ 11151us) server_negotiation.cc:407] Connection header received
0619 01:02:40.052708 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.052714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.052783 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.052878 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.054562 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.055751 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.057843 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.058572 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.058691 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.064757 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.064767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.064774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.064822 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.069079 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.069860 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.069865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.069869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.070044 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.070465 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.070470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.070473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.070847 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.071081 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.071445 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.071654 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
W20260619 01:02:40.075222 14035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:02:40.075341 14035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:02:40.072355 13943 master_runner.cc:186] Error getting master registration for 127.0.97.62:33893: OK, instance_id { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" instance_seqno: 1781830958989273 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:02:40.081027 13943 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:02:40.123734 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 4
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830958, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:02:40.138438 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.124304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51100)
0619 01:02:40.124506 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:40.124527 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:40.124653 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:40.124969 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:40.124977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:40.125482 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:40.125711 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.125723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.126969 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.126975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.128695 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.128708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.129427 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.129433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.129618 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.130283 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:40.130300 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:40.132488 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:40.136251 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.136255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.136258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.136606 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.136988 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.136992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.136994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.137112 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.137711 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:40.137714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:40.137825 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 01:02:40.137930 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:02:40.138161 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 01:02:40.138653 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.124393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51100 (local address 127.0.97.60:41909)
0619 01:02:40.124571 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.124576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.124594 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:02:40.124768 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.124771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.124826 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.124952 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.125845 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.126775 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.128842 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.129277 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.129420 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.132662 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.132669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.132674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.132709 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.136138 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.136712 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.136717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.136721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.136883 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.137247 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.137252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.137256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.137587 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.137735 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.138052 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.138272 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:02:40.143618 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: Bootstrap starting.
I20260619 01:02:40.146596 13862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:40.147665 13862 log.cc:824] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:40.150278 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: No bootstrap required, opened a new log
I20260619 01:02:40.159054 13862 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.159493 13862 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:40.159612 13862 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb3e17933f24780a2cb18ae91906f3c, State: Initialized, Role: FOLLOWER
I20260619 01:02:40.160284 13862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.160902 14024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:02:40.161197 14024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.161253 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.161612 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.162580 13862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:40.162899 14024 leader_election.cc:288] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb3e17933f24780a2cb18ae91906f3c (127.0.97.62:33893), 782f40f3d53e4e36854d377d22118e1a (127.0.97.61:34839)
I20260619 01:02:40.163556 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d42278ca3e843e29282266cc9da0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb3e17933f24780a2cb18ae91906f3c" is_pre_election: true
I20260619 01:02:40.164068 13837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d42278ca3e843e29282266cc9da0580 in term 0.
I20260619 01:02:40.164810 13962 leader_election.cc:302] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42278ca3e843e29282266cc9da0580, ccb3e17933f24780a2cb18ae91906f3c; no voters:
I20260619 01:02:40.165354 14024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:02:40.165551 14024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:02:40.165691 14024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:40.175820 14024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.179081 14024 leader_election.cc:288] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [CANDIDATE]: Term 1 election: Requested vote from peers ccb3e17933f24780a2cb18ae91906f3c (127.0.97.62:33893), 782f40f3d53e4e36854d377d22118e1a (127.0.97.61:34839)
I20260619 01:02:40.179772 13837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d42278ca3e843e29282266cc9da0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb3e17933f24780a2cb18ae91906f3c"
I20260619 01:02:40.179989 13837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:40.184178 13837 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d42278ca3e843e29282266cc9da0580 in term 1.
I20260619 01:02:40.184739 13962 leader_election.cc:302] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d42278ca3e843e29282266cc9da0580, ccb3e17933f24780a2cb18ae91906f3c; no voters:
I20260619 01:02:40.185130 14024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:40.191183 13936 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } attempt: 5
W20260619 01:02:40.195458 13964 tablet.cc:2369] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:40.179637 13795 master_runner.cc:186] Error getting master registration for 127.0.97.61:34839: OK, instance_id { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" instance_seqno: 1781830959466927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:02:40.201048 13795 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 01:02:40.201654 14049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:02:40.201833 14049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:02:40.202039 14024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [term 1 LEADER]: Becoming Leader. State: Replica: 0d42278ca3e843e29282266cc9da0580, State: Running, Role: LEADER
Jun 19 01:02:40 dist-test-slave-3rxd krb5kdc[13782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:02:40.203514 14024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.213897 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.192529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.97.60:41909)
0619 01:02:40.192716 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:40.192722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:40.192741 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:02:40.192797 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:40.192805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:40.192883 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:40.192978 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:40.194399 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.195571 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.197445 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.198103 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.198224 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.203564 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:40.203575 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:40.203580 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:40.203617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:40.207498 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.208712 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.208718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.208721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.208859 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:40.209718 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:40.209745 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:40.209751 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:40.210244 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:40.210445 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:40.211857 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0619 01:02:40.212571 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:02:40.215930 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:40.191800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41909 (local address 127.0.0.1:51114)
0619 01:02:40.192171 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:40.192194 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:40.192316 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:40.193157 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:40.193162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:40.193695 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:40.193930 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.193941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.195815 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.195821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.197266 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:40.197284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:40.198435 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:40.198441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:40.198664 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:40.199457 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:40.199484 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:40.203374 (+ 3890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:40.207634 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.207669 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.207673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.208531 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.209060 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:40.209091 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:40.209095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:40.209410 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:40.210586 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:40.210594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:40.211021 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 01:02:40.214314 (+ 3293us) client_negotiation.cc:240] Negotiation successful
0619 01:02:40.215169 (+ 855us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260619 01:02:40.218120 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d42278ca3e843e29282266cc9da0580. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.218508 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:40.221927 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: Bootstrap starting.
I20260619 01:02:40.223680 14053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:40.226257 14053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:40.227766 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: Neither blocks nor log segments found. Creating new log.
W20260619 01:02:40.228955 13816 tablet.cc:2369] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:40.231300 13936 log.cc:824] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:40.248713 13837 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [term 1 FOLLOWER]: Refusing update from remote peer 0d42278ca3e843e29282266cc9da0580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:40.249434 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: No bootstrap required, opened a new log
I20260619 01:02:40.253086 14051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:40.269658 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.270385 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:40.270606 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782f40f3d53e4e36854d377d22118e1a, State: Initialized, Role: FOLLOWER
I20260619 01:02:40.271713 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } }
I20260619 01:02:40.275259 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d42278ca3e843e29282266cc9da0580. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.275583 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.279258 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.279685 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.280354 14054 mvcc.cc:205] Tried to move back new op lower bound from 7298379613157306368 to 7298379613012996096. Current Snapshot: MvccSnapshot[applied={T|T < 7298379613157306368}]
I20260619 01:02:40.282881 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:40.283252 13909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:40.286834 13911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d42278ca3e843e29282266cc9da0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782f40f3d53e4e36854d377d22118e1a" is_pre_election: true
I20260619 01:02:40.288700 13910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d42278ca3e843e29282266cc9da0580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782f40f3d53e4e36854d377d22118e1a"
I20260619 01:02:40.290980 13909 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [term 1 FOLLOWER]: Refusing update from remote peer 0d42278ca3e843e29282266cc9da0580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:40.291930 14024 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:40.303843 14056 mvcc.cc:205] Tried to move back new op lower bound from 7298379613157306368 to 7298379613012996096. Current Snapshot: MvccSnapshot[applied={T|T < 7298379613157306368 or (T in {7298379613157306368})}]
I20260619 01:02:40.304733 14038 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.304973 14038 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.312809 14024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.313256 14024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:40.316890 14053 catalog_manager.cc:1347] Generated new cluster ID: 58e8721e728d4851b418bb8945976cad
I20260619 01:02:40.317026 14053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:40.340862 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d42278ca3e843e29282266cc9da0580. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.341166 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.341470 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.341787 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:02:40.375372 14051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d42278ca3e843e29282266cc9da0580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccb3e17933f24780a2cb18ae91906f3c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33893 } } peers { permanent_uuid: "782f40f3d53e4e36854d377d22118e1a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34839 } } peers { permanent_uuid: "0d42278ca3e843e29282266cc9da0580" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41909 } } }
I20260619 01:02:40.383671 14051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:40.385552 14053 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:02:40.388208 14053 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:40.416049 14053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d42278ca3e843e29282266cc9da0580: Generated new TSK 0
I20260619 01:02:40.417520 14053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:02:40.455796 14077 catalog_manager.cc:1259] Loaded cluster ID: 58e8721e728d4851b418bb8945976cad
I20260619 01:02:40.459300 14077 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: loading cluster ID for follower catalog manager: success
I20260619 01:02:40.464447 14077 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: acquiring CA information for follower catalog manager: success
I20260619 01:02:40.478430 14077 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 782f40f3d53e4e36854d377d22118e1a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:02:41.203843 14049 catalog_manager.cc:1259] Loaded cluster ID: 58e8721e728d4851b418bb8945976cad
I20260619 01:02:41.203945 14049 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: loading cluster ID for follower catalog manager: success
I20260619 01:02:41.206324 14049 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: acquiring CA information for follower catalog manager: success
I20260619 01:02:41.208082 14049 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccb3e17933f24780a2cb18ae91906f3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:02:41.575538 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13795
I20260619 01:02:41.576686 13857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:41.686641 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13795
I20260619 01:02:41.707350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13869
I20260619 01:02:41.708460 13931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:41.815702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13869
I20260619 01:02:41.835613 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 13943
I20260619 01:02:41.836663 14005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:41.947693 14088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:41.946922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33893 (local address 127.0.0.1:55734)
0619 01:02:41.947386 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:41.947522 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":259,"threads_started":1}
I20260619 01:02:41.947685 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 13943
2026-06-19T01:02:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4661 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:02:42 dist-test-slave-3rxd krb5kdc[14091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:02:42 dist-test-slave-3rxd krb5kdc[14091](info): set up 2 sockets
Jun 19 01:02:42 dist-test-slave-3rxd krb5kdc[14091](info): commencing operation
krb5kdc: starting...
W20260619 01:02:43.031229 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:02:43.248028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33869
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:43.563210 14104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:43.563468 14104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:43.563563 14104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:43.570833 14104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:43.570935 14104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:43.570987 14104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:43.571043 14104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:43.571087 14104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:43.580381 14104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:43.581503 14104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:43.582667 14104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:43.594056 14109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:43.594142 14110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:43.594091 14112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:43.594933 14104 server_base.cc:1034] running on GCE node
I20260619 01:02:43.595614 14104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:43.596881 14104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:43.598093 14104 hybrid_clock.cc:648] HybridClock initialized: now 1781830963598019 us; error 54 us; skew 500 ppm
I20260619 01:02:43.598429 14104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:43.601374 14104 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:43.602803 14104 webserver.cc:460] Webserver started at http://127.0.97.62:44353/ using document root <none> and password file <none>
I20260619 01:02:43.603443 14104 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:43.603560 14104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:43.603890 14104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:43.606698 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "83bd3fb07fd0499899c7f29da5d91ccf"
format_stamp: "Formatted at 2026-06-19 01:02:43 on dist-test-slave-3rxd"
server_key: "7eedf0c9a23e66ee4c9250859d870202"
server_key_iv: "b5ec737c19f4485b2e64701a50b64dd2"
server_key_version: "encryptionkey@0"
I20260619 01:02:43.607533 14104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "83bd3fb07fd0499899c7f29da5d91ccf"
format_stamp: "Formatted at 2026-06-19 01:02:43 on dist-test-slave-3rxd"
server_key: "7eedf0c9a23e66ee4c9250859d870202"
server_key_iv: "b5ec737c19f4485b2e64701a50b64dd2"
server_key_version: "encryptionkey@0"
I20260619 01:02:43.612360 14104 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 01:02:43.615743 14119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:43.616988 14104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:02:43.617156 14104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "83bd3fb07fd0499899c7f29da5d91ccf"
format_stamp: "Formatted at 2026-06-19 01:02:43 on dist-test-slave-3rxd"
server_key: "7eedf0c9a23e66ee4c9250859d870202"
server_key_iv: "b5ec737c19f4485b2e64701a50b64dd2"
server_key_version: "encryptionkey@0"
I20260619 01:02:43.617338 14104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:43.642530 14104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:43.646260 14104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:43.646477 14104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:43.664628 14104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33869
I20260619 01:02:43.664666 14170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33869 every 8 connection(s)
I20260619 01:02:43.665941 14104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:43.668571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14104
I20260619 01:02:43.668767 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:02:43.669384 388 external_mini_cluster.cc:1428] Setting key 54c7dae388144cc466b87aafb7ad2828
I20260619 01:02:43.669456 14171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:43.683303 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf: Bootstrap starting.
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:43.686251 14171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:43.687304 14171 log.cc:824] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:43.689821 14171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf: No bootstrap required, opened a new log
I20260619 01:02:43.692207 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:43.671097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.97.62:33869)
0619 01:02:43.671875 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:43.671885 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:43.671922 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:02:43.673774 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:43.673793 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:43.674131 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:43.674423 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:43.676239 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:43.677461 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:43.679298 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:43.679989 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:43.680119 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:43.685124 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:43.685132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:43.685144 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:43.685176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:43.688339 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:43.689009 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:43.689015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:43.689020 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:43.689168 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:43.689551 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:43.689559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:43.689561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:43.690057 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:43.690394 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:43.690833 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 01:02:43.691221 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260619 01:02:43.695708 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:02:43.697729 14171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } }
I20260619 01:02:43.698061 14171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:43.698189 14171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83bd3fb07fd0499899c7f29da5d91ccf, State: Initialized, Role: FOLLOWER
I20260619 01:02:43.698772 14171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } }
I20260619 01:02:43.699012 14171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:43.699139 14171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:43.699294 14171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:43.701243 14171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } }
I20260619 01:02:43.701792 14171 leader_election.cc:302] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83bd3fb07fd0499899c7f29da5d91ccf; no voters:
I20260619 01:02:43.702365 14171 leader_election.cc:288] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:02:43.702499 14176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:43.703224 14176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [term 1 LEADER]: Becoming Leader. State: Replica: 83bd3fb07fd0499899c7f29da5d91ccf, State: Running, Role: LEADER
I20260619 01:02:43.703802 14176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } }
I20260619 01:02:43.704288 14171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:43.706241 14178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83bd3fb07fd0499899c7f29da5d91ccf. Latest consensus state: current_term: 1 leader_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } } }
I20260619 01:02:43.706594 14178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:43.707257 14177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83bd3fb07fd0499899c7f29da5d91ccf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33869 } } }
I20260619 01:02:43.707652 14177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:43.707863 14182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:43.714978 14182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:43.725443 14182 catalog_manager.cc:1347] Generated new cluster ID: 762aebe824af4b31b916927161733af5
I20260619 01:02:43.725541 14182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:43.751996 14182 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:02:43.753268 14182 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:43.763685 14182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83bd3fb07fd0499899c7f29da5d91ccf: Generated new TSK 0
I20260619 01:02:43.764425 14182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:02:43.806627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14104
I20260619 01:02:43.807735 14166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:43.901597 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14104
2026-06-19T01:02:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1949 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14200](info): set up 2 sockets
Jun 19 01:02:43 dist-test-slave-3rxd krb5kdc[14200](info): commencing operation
krb5kdc: starting...
W20260619 01:02:44.979383 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:02:45 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:02:45.194487 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:45.505620 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:45.505868 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:45.506021 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:45.514183 14213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:45.514303 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:45.514384 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:45.514515 14213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:45.514583 14213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:45.524621 14213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:45.525820 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:45.527099 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:45.538010 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:45.538137 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:45.538326 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:45.538856 14213 server_base.cc:1034] running on GCE node
I20260619 01:02:45.539652 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:45.540974 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:45.542196 14213 hybrid_clock.cc:648] HybridClock initialized: now 1781830965542130 us; error 53 us; skew 500 ppm
I20260619 01:02:45.542551 14213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:45 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:45.545742 14213 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:45.547164 14213 webserver.cc:460] Webserver started at http://127.0.97.62:44305/ using document root <none> and password file <none>
I20260619 01:02:45.547801 14213 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:45.547907 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:45.548146 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:45.551080 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "897701f45a4c412f8cb2f4bb0900eb91"
format_stamp: "Formatted at 2026-06-19 01:02:45 on dist-test-slave-3rxd"
server_key: "ae54e33bd424bc6a86d74f8344b1c761"
server_key_iv: "ca398767a7d67ec3b582b58063c41b54"
server_key_version: "encryptionkey@0"
I20260619 01:02:45.551813 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "897701f45a4c412f8cb2f4bb0900eb91"
format_stamp: "Formatted at 2026-06-19 01:02:45 on dist-test-slave-3rxd"
server_key: "ae54e33bd424bc6a86d74f8344b1c761"
server_key_iv: "ca398767a7d67ec3b582b58063c41b54"
server_key_version: "encryptionkey@0"
I20260619 01:02:45.556582 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:02:45.559795 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:45.560926 14213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260619 01:02:45.561087 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "897701f45a4c412f8cb2f4bb0900eb91"
format_stamp: "Formatted at 2026-06-19 01:02:45 on dist-test-slave-3rxd"
server_key: "ae54e33bd424bc6a86d74f8344b1c761"
server_key_iv: "ca398767a7d67ec3b582b58063c41b54"
server_key_version: "encryptionkey@0"
I20260619 01:02:45.561259 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:45.578778 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:45.582363 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:45.582629 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:45.600361 14213 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39657
I20260619 01:02:45.600399 14279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39657 every 8 connection(s)
I20260619 01:02:45.601552 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:45.604843 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14213
I20260619 01:02:45.605046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:02:45.605065 14280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:45.605695 388 external_mini_cluster.cc:1428] Setting key 847ec911fe0e9640acfd65a96e9bed4b
I20260619 01:02:45.619139 14280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap starting.
Jun 19 01:02:45 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:45.622087 14280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:45.623219 14280 log.cc:824] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:45.625857 14280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: No bootstrap required, opened a new log
I20260619 01:02:45.628230 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:45.607544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.0.97.62:39657)
0619 01:02:45.608335 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:45.608345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:45.608383 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:02:45.609803 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:45.609822 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:45.610175 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:45.610508 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:45.612253 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:45.613491 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:45.615333 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:45.615984 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:45.616116 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:45.620937 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:45.620945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:45.620971 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:45.621014 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:45.624411 (+ 3397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:45.625066 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:45.625072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:45.625077 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:45.625245 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:45.625639 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:45.625643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:45.625645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:45.626172 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:45.626483 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:45.626941 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 01:02:45.627330 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":381,"threads_started":1}
I20260619 01:02:45.633841 14280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:45.634181 14280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:45.634286 14280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Initialized, Role: FOLLOWER
I20260619 01:02:45.634832 14280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:45.635058 14280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:45.635210 14280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:45.635382 14280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:45.637497 14280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:45.637941 14280 leader_election.cc:302] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897701f45a4c412f8cb2f4bb0900eb91; no voters:
I20260619 01:02:45.638490 14280 leader_election.cc:288] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:02:45.638576 14285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:45.640445 14285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 LEADER]: Becoming Leader. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Running, Role: LEADER
I20260619 01:02:45.640657 14280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:45.640929 14285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:45.643642 14286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:45.643712 14287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897701f45a4c412f8cb2f4bb0900eb91. Latest consensus state: current_term: 1 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:45.644152 14287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:45.644148 14286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:45.645906 14295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:45.651471 14295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:45.659533 14295 catalog_manager.cc:1347] Generated new cluster ID: 26bc6081faf343629c9df14e4374f88a
I20260619 01:02:45.659628 14295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:45.681933 14295 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:02:45.683722 14295 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:45.691072 14295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Generated new TSK 0
I20260619 01:02:45.692042 14295 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:02:45.742969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:46.061997 14306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:46.062288 14306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:46.062412 14306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:46.069360 14306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:46.069465 14306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:46.069586 14306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:02:46.078668 14306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:46.079891 14306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:46.081118 14306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:46.093686 14314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:46.093745 14311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:46.093621 14312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:46.094525 14306 server_base.cc:1034] running on GCE node
I20260619 01:02:46.095122 14306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:46.095984 14306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:46.097208 14306 hybrid_clock.cc:648] HybridClock initialized: now 1781830966097173 us; error 55 us; skew 500 ppm
I20260619 01:02:46.097522 14306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:46 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:46.100576 14306 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:02:46.101955 14306 webserver.cc:460] Webserver started at http://127.0.97.1:44345/ using document root <none> and password file <none>
I20260619 01:02:46.102581 14306 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:46.102675 14306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:46.103029 14306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:46.105702 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "4e588cbac6ec47a293bc594294122999"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "20fd384bc4853622d84222bd3ff47f95"
server_key_iv: "2803bd0a7cbfa6ccfba4cd2ce349a2b3"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.106400 14306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "4e588cbac6ec47a293bc594294122999"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "20fd384bc4853622d84222bd3ff47f95"
server_key_iv: "2803bd0a7cbfa6ccfba4cd2ce349a2b3"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.111167 14306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 01:02:46.114318 14321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.115577 14306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:02:46.115705 14306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "4e588cbac6ec47a293bc594294122999"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "20fd384bc4853622d84222bd3ff47f95"
server_key_iv: "2803bd0a7cbfa6ccfba4cd2ce349a2b3"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.115860 14306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:46.134022 14306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:46.137600 14306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:46.137838 14306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:46.138666 14306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:46.139837 14306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:46.139902 14306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.139976 14306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:46.140019 14306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.171983 14306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44171
I20260619 01:02:46.172070 14433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44171 every 8 connection(s)
I20260619 01:02:46.173298 14306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:02:46.174185 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14306
I20260619 01:02:46.174401 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:02:46.175275 388 external_mini_cluster.cc:1428] Setting key 0ad71261eeaf1c08f268089715de55bf
Jun 19 01:02:46 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:46.196617 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:46.176225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40167 (local address 127.0.97.62:39657)
0619 01:02:46.176690 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:46.176694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:46.177908 (+ 1214us) server_negotiation.cc:407] Connection header received
0619 01:02:46.180757 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:46.180761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:46.180825 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:46.180917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:46.183091 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.183821 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.186193 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.186599 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.186792 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:46.190227 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:46.190232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:46.190235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:46.190267 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:46.193190 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:46.194078 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:46.194084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:46.194090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:46.194232 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:46.194687 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:46.194698 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:46.194700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:46.195046 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:46.195235 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:46.195943 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 01:02:46.196183 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":244,"threads_started":1}
I20260619 01:02:46.197551 14437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:46.176829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:40167)
0619 01:02:46.177702 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:46.177765 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:46.180472 (+ 2707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:46.181219 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:46.181233 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:46.182039 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:46.182901 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:46.182931 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.183982 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.183989 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:46.186039 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:46.186051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.186903 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.186913 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:46.187158 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:46.187861 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:46.187888 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:46.189967 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:46.193368 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:46.193379 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:46.193395 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:46.193923 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:46.194352 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:46.194358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:46.194362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:46.194557 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:46.195191 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:46.195199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:46.195564 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 01:02:46.195845 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 01:02:46.196360 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":349,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:02:46.199335 14434 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:46.199641 14434 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:46.200438 14434 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:02:46.202673 14245 ts_manager.cc:194] Registered new tserver with Master: 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171)
I20260619 01:02:46.204094 14245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:02:46.226473 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:46.537398 14440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:46.537716 14440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:46.537905 14440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:46.545261 14440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:46.545403 14440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:46.545588 14440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:02:46.554857 14440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:46.556051 14440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:46.557317 14440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:46.569576 14448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:46.569648 14445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:46.569623 14446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:46.570818 14440 server_base.cc:1034] running on GCE node
I20260619 01:02:46.571379 14440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:46.572229 14440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:46.573447 14440 hybrid_clock.cc:648] HybridClock initialized: now 1781830966573406 us; error 54 us; skew 500 ppm
I20260619 01:02:46.573745 14440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:46 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:46.576740 14440 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:02:46.578027 14440 webserver.cc:460] Webserver started at http://127.0.97.2:45703/ using document root <none> and password file <none>
I20260619 01:02:46.578622 14440 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:46.578774 14440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:46.579034 14440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:46.581584 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "1467366a6b679d5d09bf39fd438e5712"
server_key_iv: "42806196fa0d5519ed76ff05b3f9f7cb"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.582278 14440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "1467366a6b679d5d09bf39fd438e5712"
server_key_iv: "42806196fa0d5519ed76ff05b3f9f7cb"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.586993 14440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.007s
I20260619 01:02:46.589991 14455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.591280 14440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:02:46.591439 14440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "1467366a6b679d5d09bf39fd438e5712"
server_key_iv: "42806196fa0d5519ed76ff05b3f9f7cb"
server_key_version: "encryptionkey@0"
I20260619 01:02:46.591642 14440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:46.636772 14440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:46.640784 14440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:46.641069 14440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:46.642104 14440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:46.643503 14440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:46.643575 14440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.643743 14440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:46.643791 14440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:46.677982 14440 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40235
I20260619 01:02:46.678025 14567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40235 every 8 connection(s)
I20260619 01:02:46.679281 14440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:02:46.685257 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14440
I20260619 01:02:46.685453 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:02:46.685981 388 external_mini_cluster.cc:1428] Setting key 3e4d1c40414db777239513d769a47d38
Jun 19 01:02:46 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:46.699222 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:46.681670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35335 (local address 127.0.97.62:39657)
0619 01:02:46.681870 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:46.681876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:46.683103 (+ 1227us) server_negotiation.cc:407] Connection header received
0619 01:02:46.685006 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:46.685011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:46.685094 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:46.685182 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:46.687058 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.687912 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.689583 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.689948 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.690082 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:46.693371 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:46.693376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:46.693379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:46.693409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:46.696033 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:46.696902 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:46.696905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:46.696908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:46.697027 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:46.697502 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:46.697505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:46.697507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:46.697769 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:46.697871 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:46.698822 (+ 951us) server_negotiation.cc:299] Negotiation successful
0619 01:02:46.698985 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:02:46.700651 14570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:46.682212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:35335)
0619 01:02:46.682930 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:46.682970 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:46.684656 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:46.685464 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:46.685478 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:46.686186 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:46.686887 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:46.686907 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.688068 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.688078 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:46.689436 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:46.689448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:46.690168 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:46.690175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:46.690430 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:46.691114 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:46.691136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:46.693163 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:46.696178 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:46.696189 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:46.696206 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:46.696791 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:46.697135 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:46.697142 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:46.697148 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:46.697378 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:46.697892 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:46.697902 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:46.698367 (+ 465us) client_negotiation.cc:769] Sending connection context
0619 01:02:46.698652 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 01:02:46.699219 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":263,"threads_started":1}
I20260619 01:02:46.702132 14568 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:46.702412 14568 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:46.703184 14568 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260619 01:02:46.704602 14245 ts_manager.cc:194] Registered new tserver with Master: a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235)
I20260619 01:02:46.705403 14245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35335
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:02:46.736281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:47.049254 14573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:47.049499 14573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:47.049585 14573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:47.056399 14573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:47.056489 14573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:47.056599 14573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:02:47.065275 14573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:47.066306 14573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:47.067469 14573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:47.079502 14578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:47.079648 14579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:47.079917 14581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:47.080662 14573 server_base.cc:1034] running on GCE node
I20260619 01:02:47.081221 14573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:47.082056 14573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:47.083237 14573 hybrid_clock.cc:648] HybridClock initialized: now 1781830967083169 us; error 65 us; skew 500 ppm
I20260619 01:02:47.083524 14573 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:47 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:47.086479 14573 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:02:47.087963 14573 webserver.cc:460] Webserver started at http://127.0.97.3:35157/ using document root <none> and password file <none>
I20260619 01:02:47.088599 14573 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:47.088749 14573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:47.089035 14573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:02:47.091920 14573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "3b6e5549b7bb494eba94ac967cd284fc"
format_stamp: "Formatted at 2026-06-19 01:02:47 on dist-test-slave-3rxd"
server_key: "853a84e74c465245af2b2976aef69f47"
server_key_iv: "da237d2556229fad108d1d43246b2422"
server_key_version: "encryptionkey@0"
I20260619 01:02:47.092669 14573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3b6e5549b7bb494eba94ac967cd284fc"
format_stamp: "Formatted at 2026-06-19 01:02:47 on dist-test-slave-3rxd"
server_key: "853a84e74c465245af2b2976aef69f47"
server_key_iv: "da237d2556229fad108d1d43246b2422"
server_key_version: "encryptionkey@0"
I20260619 01:02:47.097636 14573 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 01:02:47.100744 14588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:47.101843 14573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 01:02:47.101998 14573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "3b6e5549b7bb494eba94ac967cd284fc"
format_stamp: "Formatted at 2026-06-19 01:02:47 on dist-test-slave-3rxd"
server_key: "853a84e74c465245af2b2976aef69f47"
server_key_iv: "da237d2556229fad108d1d43246b2422"
server_key_version: "encryptionkey@0"
I20260619 01:02:47.102178 14573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:47.132122 14573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:47.135669 14573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:47.135980 14573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:47.136793 14573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:47.137977 14573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:47.138077 14573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:47.138217 14573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:47.138298 14573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:47.170279 14573 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45965
I20260619 01:02:47.170359 14700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45965 every 8 connection(s)
I20260619 01:02:47.171617 14573 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:02:47.176550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14573
I20260619 01:02:47.176764 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:02:47.177196 388 external_mini_cluster.cc:1428] Setting key af10aecd666c786f8501035c84dcb56d
Jun 19 01:02:47 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:47.192123 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.174130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41151 (local address 127.0.97.62:39657)
0619 01:02:47.174354 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.174361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.175578 (+ 1217us) server_negotiation.cc:407] Connection header received
0619 01:02:47.178008 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.178015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.178110 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.178242 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:47.179824 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.180639 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.182262 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.182701 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.182902 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.186448 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:47.186453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:47.186457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:47.186491 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:47.188982 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.189661 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.189665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.189667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.189788 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.190202 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.190207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.190220 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.190551 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:47.190712 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.191546 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.191786 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:02:47.193205 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.174530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:41151)
0619 01:02:47.175386 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.175438 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.177702 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.178479 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.178487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.179085 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:47.179649 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.179667 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.180828 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.180836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.182105 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.182118 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.182984 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.182995 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.183267 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.184011 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:47.184035 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:47.186245 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:47.189104 (+ 2859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:47.189111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:47.189122 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:47.189537 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:47.189885 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:47.189889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:47.189891 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:47.190094 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:47.190696 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:47.190754 (+ 58us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:47.191087 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.191426 (+ 339us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.191947 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":372,"threads_started":1}
I20260619 01:02:47.194594 14701 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:47.194953 14701 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:47.195887 14701 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260619 01:02:47.197288 14245 ts_manager.cc:194] Registered new tserver with Master: 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965)
I20260619 01:02:47.198102 14245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41151
I20260619 01:02:47.204628 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:02:47.206849 14434 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:47.220885 14436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.209441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.0.97.62:39657)
0619 01:02:47.209589 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.209593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.210077 (+ 484us) server_negotiation.cc:407] Connection header received
0619 01:02:47.210225 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.210229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.210283 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.210357 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:47.211668 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.212342 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.213647 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.214048 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.214148 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.216374 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:47.216378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:47.216381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:47.216409 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:47.218756 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.219372 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.219375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.219377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.219479 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.219776 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.219779 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.219781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.220054 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:47.220277 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.220518 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.220673 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:02:47.224438 14245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:02:47.231081 14245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:02:47.267827 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.243074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.0.97.2:40235)
0619 01:02:47.256143 (+ 13069us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.256155 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.260607 (+ 4452us) server_negotiation.cc:407] Connection header received
0619 01:02:47.260827 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.260834 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.261012 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.261210 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:47.261934 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.263264 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.265737 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.266670 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.266863 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.267065 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.267350 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.267508 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12896,"thread_start_us":12784,"threads_started":1}
I20260619 01:02:47.267829 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.242509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40235 (local address 127.0.0.1:41816)
0619 01:02:47.260463 (+ 17954us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.260495 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.260636 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.261218 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.261224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.261250 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:47.261722 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.261734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.263398 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.263404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.265548 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.265566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.266930 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.266937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.267140 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.267174 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.267303 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.267485 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17839,"thread_start_us":16395,"threads_started":1}
I20260619 01:02:47.268266 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.241870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45965 (local address 127.0.0.1:51998)
0619 01:02:47.258386 (+ 16516us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.258446 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.258669 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.261270 (+ 2601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.261274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.261299 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:47.261721 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.261730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.263224 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.263231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.265352 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.265364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.267467 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.267473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.267645 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.267677 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.267757 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.267878 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16195,"thread_start_us":18346,"threads_started":1}
I20260619 01:02:47.269184 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.242235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51998 (local address 127.0.97.3:45965)
0619 01:02:47.260825 (+ 18590us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.260831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.260854 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:02:47.260929 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.260933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.261093 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.261283 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:47.261896 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.263078 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.265537 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.266983 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.267110 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.267339 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.268693 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.268843 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18437,"thread_start_us":18282,"threads_started":1}
I20260619 01:02:47.271425 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.242169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52598)
0619 01:02:47.260027 (+ 17858us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.260048 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.260211 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.262200 (+ 1989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.262204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.262226 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:47.262566 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.262576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.264405 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.264409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.266998 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.267011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.269067 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.269073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.270897 (+ 1824us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.270929 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.271019 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.271142 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17688,"thread_start_us":17648,"threads_started":1}
I20260619 01:02:47.271673 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.242800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.0.97.1:44171)
0619 01:02:47.252522 (+ 9722us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.252536 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.260162 (+ 7626us) server_negotiation.cc:407] Connection header received
0619 01:02:47.261558 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.261568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.261788 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.261930 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:47.262788 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.264255 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.267170 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.268874 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.268997 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.269223 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.271094 (+ 1871us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.271267 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9483,"thread_start_us":9320,"threads_started":1}
I20260619 01:02:47.271708 14503 tablet_service.cc:1467] Processing CreateTablet for tablet d35a821d172e45b3993320e0789154aa (DEFAULT_TABLE table=test-table [id=f7acf607432749c4836b7c6094a6bd8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:47.272390 14636 tablet_service.cc:1467] Processing CreateTablet for tablet d35a821d172e45b3993320e0789154aa (DEFAULT_TABLE table=test-table [id=f7acf607432749c4836b7c6094a6bd8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:47.273188 14503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35a821d172e45b3993320e0789154aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:47.273627 14636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35a821d172e45b3993320e0789154aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:47.275120 14369 tablet_service.cc:1467] Processing CreateTablet for tablet d35a821d172e45b3993320e0789154aa (DEFAULT_TABLE table=test-table [id=f7acf607432749c4836b7c6094a6bd8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:47.276345 14369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d35a821d172e45b3993320e0789154aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:47.284271 14719 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Bootstrap starting.
I20260619 01:02:47.287530 14719 tablet_bootstrap.cc:654] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:47.287519 14720 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Bootstrap starting.
I20260619 01:02:47.288928 14719 log.cc:824] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:47.290270 14721 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Bootstrap starting.
I20260619 01:02:47.291308 14720 tablet_bootstrap.cc:654] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:47.292598 14720 log.cc:824] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:47.293618 14721 tablet_bootstrap.cc:654] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:47.293658 14719 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: No bootstrap required, opened a new log
I20260619 01:02:47.293936 14719 ts_tablet_manager.cc:1397] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 01:02:47.294651 14721 log.cc:824] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:47.295526 14720 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: No bootstrap required, opened a new log
I20260619 01:02:47.295776 14720 ts_tablet_manager.cc:1397] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 01:02:47.297271 14721 tablet_bootstrap.cc:492] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: No bootstrap required, opened a new log
I20260619 01:02:47.297536 14721 ts_tablet_manager.cc:1397] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 01:02:47.304013 14721 raft_consensus.cc:348] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.304543 14721 raft_consensus.cc:374] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:47.304353 14719 raft_consensus.cc:348] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.304700 14721 raft_consensus.cc:729] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e588cbac6ec47a293bc594294122999, State: Initialized, Role: FOLLOWER
I20260619 01:02:47.304847 14719 raft_consensus.cc:374] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:47.304996 14719 raft_consensus.cc:729] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6e5549b7bb494eba94ac967cd284fc, State: Initialized, Role: FOLLOWER
I20260619 01:02:47.305483 14721 consensus_queue.cc:260] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.305743 14719 consensus_queue.cc:260] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.305955 14720 raft_consensus.cc:348] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.306519 14720 raft_consensus.cc:374] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:47.306700 14720 raft_consensus.cc:729] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e53dd363954afbae6c5b2b7a8585d2, State: Initialized, Role: FOLLOWER
I20260619 01:02:47.307271 14721 ts_tablet_manager.cc:1428] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 01:02:47.307574 14720 consensus_queue.cc:260] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.308596 14701 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:47.310806 14719 ts_tablet_manager.cc:1428] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260619 01:02:47.312670 14568 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:47.313076 14720 ts_tablet_manager.cc:1428] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
W20260619 01:02:47.423305 14702 tablet.cc:2369] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:02:47.425565 14435 tablet.cc:2369] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:02:47.431099 14569 tablet.cc:2369] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:47.622823 14726 raft_consensus.cc:482] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:02:47.623163 14726 raft_consensus.cc:504] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.624735 14726 leader_election.cc:288] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171), a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235)
I20260619 01:02:47.631693 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.625103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48213 (local address 127.0.97.1:44171)
0619 01:02:47.625302 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.625307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.625328 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:02:47.625634 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.625640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.625726 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.625810 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:47.626272 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.627349 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.629649 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.630888 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.631025 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.631139 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.631351 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.631485 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:02:47.631765 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.624912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.97.3:48213)
0619 01:02:47.625220 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.625246 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.625417 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.625868 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.625873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.625904 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:47.626139 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.626148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.627481 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.627484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.629508 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.629522 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.631035 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.631039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.631172 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.631195 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.631309 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.631458 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":87}
I20260619 01:02:47.632117 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.625365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43937 (local address 127.0.97.2:40235)
0619 01:02:47.625594 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.625600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.625918 (+ 318us) server_negotiation.cc:407] Connection header received
0619 01:02:47.626111 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.626116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.626173 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.626286 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:47.626777 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.627672 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.629869 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.631173 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.631311 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.631396 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.631722 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.631866 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:02:47.632392 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.624998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40235 (local address 127.0.97.3:43937)
0619 01:02:47.625771 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.625807 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:47.625982 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:47.626319 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:47.626322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:47.626343 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:47.626589 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.626604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.627806 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.627811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.629734 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:47.629743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.631373 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.631378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:47.631526 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.631562 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:02:47.631667 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:02:47.631787 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"spinlock_wait_cycles":19200,"thread_start_us":497,"threads_started":1}
I20260619 01:02:47.633088 14389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35a821d172e45b3993320e0789154aa" candidate_uuid: "3b6e5549b7bb494eba94ac967cd284fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e588cbac6ec47a293bc594294122999" is_pre_election: true
I20260619 01:02:47.633302 14523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35a821d172e45b3993320e0789154aa" candidate_uuid: "3b6e5549b7bb494eba94ac967cd284fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" is_pre_election: true
I20260619 01:02:47.633621 14389 raft_consensus.cc:2454] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6e5549b7bb494eba94ac967cd284fc in term 0.
I20260619 01:02:47.633800 14523 raft_consensus.cc:2454] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b6e5549b7bb494eba94ac967cd284fc in term 0.
I20260619 01:02:47.634310 14591 leader_election.cc:302] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6e5549b7bb494eba94ac967cd284fc, 4e588cbac6ec47a293bc594294122999; no voters:
I20260619 01:02:47.634704 14726 raft_consensus.cc:2781] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:02:47.634887 14726 raft_consensus.cc:482] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:02:47.634984 14726 raft_consensus.cc:3037] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:47.637063 14726 raft_consensus.cc:504] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.637609 14726 leader_election.cc:288] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [CANDIDATE]: Term 1 election: Requested vote from peers 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171), a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235)
I20260619 01:02:47.637945 14389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35a821d172e45b3993320e0789154aa" candidate_uuid: "3b6e5549b7bb494eba94ac967cd284fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e588cbac6ec47a293bc594294122999"
I20260619 01:02:47.638069 14523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d35a821d172e45b3993320e0789154aa" candidate_uuid: "3b6e5549b7bb494eba94ac967cd284fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
I20260619 01:02:47.638146 14389 raft_consensus.cc:3037] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:47.638283 14523 raft_consensus.cc:3037] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:47.640426 14389 raft_consensus.cc:2454] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6e5549b7bb494eba94ac967cd284fc in term 1.
I20260619 01:02:47.640502 14523 raft_consensus.cc:2454] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b6e5549b7bb494eba94ac967cd284fc in term 1.
I20260619 01:02:47.640798 14591 leader_election.cc:302] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b6e5549b7bb494eba94ac967cd284fc, 4e588cbac6ec47a293bc594294122999; no voters:
I20260619 01:02:47.641070 14726 raft_consensus.cc:2781] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:47.641639 14726 raft_consensus.cc:686] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 1 LEADER]: Becoming Leader. State: Replica: 3b6e5549b7bb494eba94ac967cd284fc, State: Running, Role: LEADER
I20260619 01:02:47.642120 14726 consensus_queue.cc:237] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:47.647256 14245 catalog_manager.cc:5495] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "3b6e5549b7bb494eba94ac967cd284fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } health_report { overall_health: UNKNOWN } } }
I20260619 01:02:47.706581 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.701125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.0.97.3:45965)
0619 01:02:47.701304 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.701308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.701323 (+ 15us) server_negotiation.cc:407] Connection header received
0619 01:02:47.701408 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.701411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.701486 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.701626 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:02:47.701971 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.702681 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.704310 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.704698 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.704807 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.705408 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:02:47.705993 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:02:47.706105 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.706265 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.706330 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:02:47.714700 14523 raft_consensus.cc:1261] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Refusing update from remote peer 3b6e5549b7bb494eba94ac967cd284fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:47.714888 14389 raft_consensus.cc:1261] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Refusing update from remote peer 3b6e5549b7bb494eba94ac967cd284fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:47.715726 14729 consensus_queue.cc:1035] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:47.716046 14726 consensus_queue.cc:1035] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:47.727030 14732 mvcc.cc:205] Tried to move back new op lower bound from 7298379643750150144 to 7298379643466096640. Current Snapshot: MvccSnapshot[applied={T|T < 7298379643750150144}]
I20260619 01:02:47.731208 14733 mvcc.cc:205] Tried to move back new op lower bound from 7298379643750150144 to 7298379643466096640. Current Snapshot: MvccSnapshot[applied={T|T < 7298379643750150144}]
I20260619 01:02:47.740427 14244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:47.740685 14244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:47.743986 14244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Sending DeleteTablet for 3 replicas of tablet d35a821d172e45b3993320e0789154aa
I20260619 01:02:47.745072 14636 tablet_service.cc:1514] Processing DeleteTablet for tablet d35a821d172e45b3993320e0789154aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51998
I20260619 01:02:47.745311 14503 tablet_service.cc:1514] Processing DeleteTablet for tablet d35a821d172e45b3993320e0789154aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41816
I20260619 01:02:47.745729 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14306
I20260619 01:02:47.745745 14369 tablet_service.cc:1514] Processing DeleteTablet for tablet d35a821d172e45b3993320e0789154aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:47 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52598
Jun 19 01:02:47 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:02:47.769907 14744 tablet_replica.cc:330] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: stopping tablet replica
I20260619 01:02:47.770571 14744 raft_consensus.cc:2229] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:47.771268 14744 raft_consensus.cc:2258] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:47.771881 14742 tablet_replica.cc:330] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: stopping tablet replica
I20260619 01:02:47.774227 14744 ts_tablet_manager.cc:1905] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:47.774756 14742 raft_consensus.cc:2229] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:02:47.775472 14742 raft_consensus.cc:2258] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:47.780243 14743 tablet_replica.cc:330] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: stopping tablet replica
I20260619 01:02:47.780385 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.747050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.0.97.1:44171)
0619 01:02:47.747278 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.747286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.755315 (+ 8029us) server_negotiation.cc:407] Connection header received
0619 01:02:47.755536 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.755545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.755659 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.755783 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:47.758313 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.759956 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.762696 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.763601 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.763713 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.770963 (+ 7250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:47.770985 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:47.771010 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:47.771063 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:47.776192 (+ 5129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.777296 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.777308 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.777314 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.777536 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.778114 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.778122 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.778128 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.778677 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:47.779033 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.779783 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.780047 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:02:47.780650 14743 raft_consensus.cc:2229] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:47.781159 14743 raft_consensus.cc:2258] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:47.781388 14742 ts_tablet_manager.cc:1905] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:47.781720 14429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:47.784596 14743 ts_tablet_manager.cc:1905] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:47.790501 14742 ts_tablet_manager.cc:1918] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:47.790735 14742 log.cc:1196] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/d35a821d172e45b3993320e0789154aa
I20260619 01:02:47.791332 14742 ts_tablet_manager.cc:1939] T d35a821d172e45b3993320e0789154aa P 3b6e5549b7bb494eba94ac967cd284fc: Deleting consensus metadata
I20260619 01:02:47.791363 14743 ts_tablet_manager.cc:1918] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:47.791517 14743 log.cc:1196] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/d35a821d172e45b3993320e0789154aa
I20260619 01:02:47.792042 14743 ts_tablet_manager.cc:1939] T d35a821d172e45b3993320e0789154aa P a0e53dd363954afbae6c5b2b7a8585d2: Deleting consensus metadata
I20260619 01:02:47.793192 14230 catalog_manager.cc:4841] TS 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965): tablet d35a821d172e45b3993320e0789154aa (table test-table [id=f7acf607432749c4836b7c6094a6bd8b]) successfully deleted
I20260619 01:02:47.794066 14232 catalog_manager.cc:4841] TS a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235): tablet d35a821d172e45b3993320e0789154aa (table test-table [id=f7acf607432749c4836b7c6094a6bd8b]) successfully deleted
I20260619 01:02:47.904474 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14306
I20260619 01:02:47.928171 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14440
W20260619 01:02:47.928141 14231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44171 (error 108)
I20260619 01:02:47.931819 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.931104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52616)
0619 01:02:47.931537 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.931628 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":226,"threads_started":1}
W20260619 01:02:47.932335 14231 catalog_manager.cc:4584] TS 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d35a821d172e45b3993320e0789154aa: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Jun 19 01:02:47 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:02:47.945950 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.929007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41826 (local address 127.0.97.2:40235)
0619 01:02:47.929783 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:47.929792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:47.929815 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:02:47.929896 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:47.929904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:47.930005 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:47.930140 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:47.931879 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.932831 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.934616 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:47.935051 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:47.935142 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:47.939980 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:47.939992 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:47.940007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:47.940037 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:47.943120 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.943833 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.943841 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.943843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.943963 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:47.944367 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:47.944371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:47.944373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:47.944754 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:47.945044 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:47.945404 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 01:02:47.945626 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607}
I20260619 01:02:47.946552 14563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:47.986188 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.985678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52620)
0619 01:02:47.985915 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.986027 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 01:02:47.987346 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:47.986934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52632)
0619 01:02:47.987103 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:47.987196 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:02:48.035005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14440
I20260619 01:02:48.047715 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.047283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52642)
0619 01:02:48.047483 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:48.047574 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 01:02:48.048784 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.048417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52644)
0619 01:02:48.048572 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:48.048662 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:02:48.058383 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14573
Jun 19 01:02:48 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:02:48.074877 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.059256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.0.97.3:45965)
0619 01:02:48.059454 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:48.059460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:48.059478 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:02:48.059600 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:48.059605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:48.059673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:48.059773 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:48.061068 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:48.061797 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:48.063166 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:48.063586 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:48.063714 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:48.068164 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:48.068183 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:48.068209 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:48.068258 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:48.072230 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:48.072852 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:48.072860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:48.072864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:48.073037 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:48.073363 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:48.073368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:48.073371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:48.073775 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:48.074071 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:48.074264 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 01:02:48.074463 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:02:48.075606 14696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:48.129578 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.129056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52654)
0619 01:02:48.129276 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:48.129382 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 01:02:48.130681 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.130223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52660)
0619 01:02:48.130445 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:48.130527 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260619 01:02:48.196993 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14573
I20260619 01:02:48.217335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14213
I20260619 01:02:48.218415 14275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:48.323912 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:48.323284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52670)
0619 01:02:48.323589 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:48.323716 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 01:02:48.324009 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14213
I20260619 01:02:48.345234 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--webserver_interface=127.0.97.62
--webserver_port=44305
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:48.655781 14752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:48.656105 14752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:48.656257 14752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:48.662925 14752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:48.663051 14752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:48.663161 14752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:48.663259 14752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:48.663329 14752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:48.675808 14752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:48.677007 14752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:48.678220 14752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:48.699747 14759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:48.699801 14758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:48.699826 14761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:48.700547 14752 server_base.cc:1034] running on GCE node
I20260619 01:02:48.701393 14752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:48.702940 14752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:48.704164 14752 hybrid_clock.cc:648] HybridClock initialized: now 1781830968704122 us; error 47 us; skew 500 ppm
I20260619 01:02:48.704546 14752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:48 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:48.710511 14752 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:48.712117 14752 webserver.cc:460] Webserver started at http://127.0.97.62:44305/ using document root <none> and password file <none>
I20260619 01:02:48.712834 14752 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:48.712992 14752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:48.718837 14752 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 01:02:48.721625 14768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:48.722962 14752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:02:48.723142 14752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "897701f45a4c412f8cb2f4bb0900eb91"
format_stamp: "Formatted at 2026-06-19 01:02:45 on dist-test-slave-3rxd"
server_key: "ae54e33bd424bc6a86d74f8344b1c761"
server_key_iv: "ca398767a7d67ec3b582b58063c41b54"
server_key_version: "encryptionkey@0"
I20260619 01:02:48.724164 14752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:48.735173 14752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:48.739164 14752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:48.739449 14752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:48.757591 14752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39657
I20260619 01:02:48.757603 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39657 every 8 connection(s)
I20260619 01:02:48.759104 14752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:48.766026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14752
I20260619 01:02:48.766618 14820 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:02:48.767127 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44171
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=44345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:48.771930 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap starting.
I20260619 01:02:48.792680 14820 log.cc:824] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:48.804418 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:02:48.804932 14820 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap complete.
I20260619 01:02:48.814209 14820 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:48.815146 14820 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Initialized, Role: FOLLOWER
I20260619 01:02:48.815801 14820 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:48.816035 14820 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:48.816210 14820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:48.816396 14820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:02:48.820269 14820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:48.820765 14820 leader_election.cc:302] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897701f45a4c412f8cb2f4bb0900eb91; no voters:
I20260619 01:02:48.821420 14820 leader_election.cc:288] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:02:48.821547 14824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:02:48.822808 14824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 LEADER]: Becoming Leader. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Running, Role: LEADER
I20260619 01:02:48.823405 14824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:48.823817 14820 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:48.826112 14826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897701f45a4c412f8cb2f4bb0900eb91. Latest consensus state: current_term: 2 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:48.826479 14826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:48.826295 14825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:48.826695 14825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:48.835600 14831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:48.845233 14831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:48.847033 14831 catalog_manager.cc:1259] Loaded cluster ID: 26bc6081faf343629c9df14e4374f88a
I20260619 01:02:48.847240 14831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:02:48.851359 14831 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 01:02:48.873737 14831 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 01:02:48.901835 14831 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:02:48.903546 14831 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Loaded TSK: 0
I20260619 01:02:48.905010 14831 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:02:49.135481 14822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:49.135793 14822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:49.135970 14822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:49.143183 14822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:49.143273 14822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:49.143425 14822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:02:49.152842 14822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44345
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:49.154057 14822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:49.155385 14822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:49.169824 14847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:49.169746 14848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:49.170496 14850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:49.170753 14822 server_base.cc:1034] running on GCE node
I20260619 01:02:49.171396 14822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:49.172349 14822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:49.173593 14822 hybrid_clock.cc:648] HybridClock initialized: now 1781830969173538 us; error 49 us; skew 500 ppm
I20260619 01:02:49.173890 14822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:49 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:49.179600 14822 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:02:49.181241 14822 webserver.cc:460] Webserver started at http://127.0.97.1:44345/ using document root <none> and password file <none>
I20260619 01:02:49.181960 14822 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:49.182135 14822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:49.187638 14822 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:02:49.190629 14857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:49.192047 14822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:02:49.192203 14822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "4e588cbac6ec47a293bc594294122999"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "20fd384bc4853622d84222bd3ff47f95"
server_key_iv: "2803bd0a7cbfa6ccfba4cd2ce349a2b3"
server_key_version: "encryptionkey@0"
I20260619 01:02:49.193096 14822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:49.211004 14822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:49.215204 14822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:49.215471 14822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:49.216379 14822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:49.217993 14864 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 01:02:49.223284 14864 ts_tablet_manager.cc:1852] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 01:02:49.224334 14864 ts_tablet_manager.cc:1905] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:49.231523 14864 ts_tablet_manager.cc:1918] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:49.231704 14864 log.cc:1196] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/d35a821d172e45b3993320e0789154aa
I20260619 01:02:49.232124 14864 ts_tablet_manager.cc:1939] T d35a821d172e45b3993320e0789154aa P 4e588cbac6ec47a293bc594294122999: Deleting consensus metadata
I20260619 01:02:49.232430 14822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 01:02:49.232575 14822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.000s sys 0.000s
I20260619 01:02:49.232702 14822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:49.232789 14822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:49.264951 14822 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44171
I20260619 01:02:49.265000 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44171 every 8 connection(s)
I20260619 01:02:49.266198 14822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:02:49.271842 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14822
I20260619 01:02:49.272565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40235
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=45703
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:49 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:49.290505 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:49.268877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57853 (local address 127.0.97.62:39657)
0619 01:02:49.269411 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:49.269422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:49.270107 (+ 685us) server_negotiation.cc:407] Connection header received
0619 01:02:49.272426 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:49.272446 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:49.272875 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:49.273257 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:49.274537 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.276538 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.278229 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.279343 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.279536 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:49.283441 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:49.283448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:49.283460 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:49.283499 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:49.286219 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:49.287006 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:49.287014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:49.287018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:49.287176 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:49.287591 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:49.287597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:49.287605 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:49.288046 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:49.288244 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:49.288964 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 01:02:49.289363 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260619 01:02:49.290503 14974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:49.269105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:57853)
0619 01:02:49.269955 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:49.269998 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:49.271828 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:49.273352 (+ 1524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:49.273359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:49.273813 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:49.274318 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:49.274334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.276870 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.276874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:49.278076 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:49.278099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.279581 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.279589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:49.279907 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:49.280648 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:49.280680 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:49.283190 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:49.286359 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:49.286369 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:49.286381 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:49.286873 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:49.287294 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:49.287299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:49.287303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:49.287475 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:49.288193 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:49.288200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:49.288511 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 01:02:49.288758 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 01:02:49.289252 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":365,"threads_started":1}
I20260619 01:02:49.293476 14971 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:49.293777 14971 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:49.294498 14971 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260619 01:02:49.296801 14785 ts_manager.cc:194] Registered new tserver with Master: 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171)
I20260619 01:02:49.298382 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57853
W20260619 01:02:49.592166 14975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:49.592407 14975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:49.592492 14975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:49.599377 14975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:49.599465 14975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:49.599566 14975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:02:49.608701 14975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45703
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:49.609764 14975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:49.610972 14975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:49.623142 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:49.623219 14983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:49.623188 14981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:49.624354 14975 server_base.cc:1034] running on GCE node
I20260619 01:02:49.624950 14975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:49.625872 14975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:49.627053 14975 hybrid_clock.cc:648] HybridClock initialized: now 1781830969627021 us; error 48 us; skew 500 ppm
I20260619 01:02:49.627352 14975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:49 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:49.630239 14975 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:02:49.631577 14975 webserver.cc:460] Webserver started at http://127.0.97.2:45703/ using document root <none> and password file <none>
I20260619 01:02:49.632218 14975 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:49.632368 14975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:49.637380 14975 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 01:02:49.639891 14990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:49.641099 14975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:02:49.641269 14975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "1467366a6b679d5d09bf39fd438e5712"
server_key_iv: "42806196fa0d5519ed76ff05b3f9f7cb"
server_key_version: "encryptionkey@0"
I20260619 01:02:49.642220 14975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:49.657094 14975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:49.661217 14975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:49.661482 14975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:49.662350 14975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:49.663661 14975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:49.663797 14975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:49.663918 14975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:49.664008 14975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:49.697665 14975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40235
I20260619 01:02:49.697763 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40235 every 8 connection(s)
I20260619 01:02:49.699281 14975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:02:49.705178 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 14975
I20260619 01:02:49.710065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45965
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=35157
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:02:49 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:49.718313 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:49.701693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54175 (local address 127.0.97.62:39657)
0619 01:02:49.701869 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:49.701873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:49.703067 (+ 1194us) server_negotiation.cc:407] Connection header received
0619 01:02:49.704854 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:49.704860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:49.704950 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:49.705057 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:49.706555 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.707398 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.708851 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.709268 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.709397 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:49.712808 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:49.712813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:49.712816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:49.712846 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:49.715402 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:49.716094 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:49.716101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:49.716104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:49.716240 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:49.716674 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:49.716681 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:49.716684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:49.716940 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:49.717088 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:49.717750 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 01:02:49.717966 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:02:49.719297 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:49.702164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:54175)
0619 01:02:49.702916 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:49.702958 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:49.704579 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:49.705306 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:49.705315 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:49.705860 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:49.706390 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:49.706406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.707541 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.707547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:49.708710 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:49.708722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:49.709475 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:49.709482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:49.709702 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:49.710412 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:49.710433 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:49.712615 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:49.715558 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:49.715567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:49.715582 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:49.715976 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:49.716373 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:49.716378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:49.716382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:49.716563 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:49.717072 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:49.717079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:49.717399 (+ 320us) client_negotiation.cc:769] Sending connection context
0619 01:02:49.717649 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 01:02:49.718149 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260619 01:02:49.720769 15103 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:49.721060 15103 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:49.721741 15103 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260619 01:02:49.723197 14785 ts_manager.cc:194] Registered new tserver with Master: a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235)
I20260619 01:02:49.724484 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54175
W20260619 01:02:50.026696 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:50.026981 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:50.027071 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:50.034183 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:50.034286 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:50.034396 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:02:50.043751 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35157
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:50.044931 15106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:50.046286 15106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:50.059095 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:50.059185 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:50.059206 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:50.059924 15106 server_base.cc:1034] running on GCE node
I20260619 01:02:50.060593 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:50.061812 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:50.063026 15106 hybrid_clock.cc:648] HybridClock initialized: now 1781830970062968 us; error 54 us; skew 500 ppm
I20260619 01:02:50.063277 15106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:50 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:50.066354 15106 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:02:50.067950 15106 webserver.cc:460] Webserver started at http://127.0.97.3:35157/ using document root <none> and password file <none>
I20260619 01:02:50.068614 15106 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:50.068758 15106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:50.073621 15106 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 01:02:50.076073 15121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:50.077298 15106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 01:02:50.077457 15106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "3b6e5549b7bb494eba94ac967cd284fc"
format_stamp: "Formatted at 2026-06-19 01:02:47 on dist-test-slave-3rxd"
server_key: "853a84e74c465245af2b2976aef69f47"
server_key_iv: "da237d2556229fad108d1d43246b2422"
server_key_version: "encryptionkey@0"
I20260619 01:02:50.078325 15106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:50.094151 15106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:50.098567 15106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:50.098869 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:50.099825 15106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:50.101146 15106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:50.101260 15106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:50.101397 15106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:50.101500 15106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:50.133908 15106 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45965
I20260619 01:02:50.133958 15233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45965 every 8 connection(s)
I20260619 01:02:50.135615 15106 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:02:50.142575 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15106
Jun 19 01:02:50 dist-test-slave-3rxd krb5kdc[14200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:02:50.156657 14973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.138263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59001 (local address 127.0.97.62:39657)
0619 01:02:50.138482 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.138489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.139705 (+ 1216us) server_negotiation.cc:407] Connection header received
0619 01:02:50.141370 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.141374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.141446 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.141551 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:50.143036 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.144139 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.146059 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.146542 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.146685 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.150009 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:50.150014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:50.150017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:50.150047 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:50.152931 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.154017 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.154021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.154023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.154213 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.154758 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.154767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.154770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.155047 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:50.155177 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.156073 (+ 896us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.156286 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:02:50.157490 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.138847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:59001)
0619 01:02:50.139547 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.139594 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.141118 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.141749 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.141756 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.142246 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:02:50.142870 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.142889 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.144299 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.144306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.145920 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.145932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.146821 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.146828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.147106 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.147743 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:02:50.147764 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:02:50.149796 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:02:50.153290 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:50.153299 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:50.153317 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:50.153886 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:50.154358 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:02:50.154363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:02:50.154367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:02:50.154604 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:02:50.155206 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:02:50.155216 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:02:50.155633 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.155917 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.156280 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260619 01:02:50.158895 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.143602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.0.97.62:39657)
0619 01:02:50.144612 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.144618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.144635 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:02:50.144681 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.144686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.144747 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.144849 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:50.146672 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.147648 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.148979 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.149488 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.149624 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.152388 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:50.152393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:50.152397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:50.152438 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:50.155736 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.156628 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.156635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.156639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.156774 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.157338 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.157343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.157345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.157705 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:50.157961 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.158305 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.158514 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":787,"threads_started":1}
I20260619 01:02:50.159323 15234 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39657
I20260619 01:02:50.159657 15234 heartbeater.cc:464] Registering TS with master...
I20260619 01:02:50.160416 15234 heartbeater.cc:511] Master 127.0.97.62:39657 requested a full tablet report, sending...
I20260619 01:02:50.161834 14785 ts_manager.cc:194] Registered new tserver with Master: 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965)
I20260619 01:02:50.162611 14785 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59001
I20260619 01:02:50.173183 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:02:50.179394 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.174113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.97.62:39657)
0619 01:02:50.174236 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.174240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.174708 (+ 468us) server_negotiation.cc:407] Connection header received
0619 01:02:50.174942 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.174945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.175061 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.175181 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:02:50.175511 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.176167 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.177514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.177917 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.178004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.178413 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:02:50.178983 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:02:50.179097 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.179176 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.179230 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 01:02:50.180171 14785 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:02:50.187513 14785 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:02:50.222967 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.198650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52672)
0619 01:02:50.211660 (+ 13010us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.211695 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.211871 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.213179 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.213190 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.213233 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:50.213744 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.213759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.217276 (+ 3517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.217285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.220311 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.220325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.221936 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.221943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.222140 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.222192 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.222337 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.222543 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12751,"thread_start_us":19389,"threads_started":1}
I20260619 01:02:50.222952 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.198849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.0.97.1:44171)
0619 01:02:50.211238 (+ 12389us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.211252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.211787 (+ 535us) server_negotiation.cc:407] Connection header received
0619 01:02:50.212069 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.212080 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.212388 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.214008 (+ 1620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:50.214274 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.217091 (+ 2817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.220454 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.221708 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.221865 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.222072 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.222382 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.222543 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12204,"thread_start_us":12090,"threads_started":1}
I20260619 01:02:50.225138 14906 tablet_service.cc:1467] Processing CreateTablet for tablet be4f641e2a17468aae748a5c91c92c0f (DEFAULT_TABLE table=test-table [id=fd7bd326fe564145884afbb5ecb7fbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:50.226498 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4f641e2a17468aae748a5c91c92c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:50.226886 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.199194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40235 (local address 127.0.0.1:41832)
0619 01:02:50.217847 (+ 18653us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.217865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.217984 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.218940 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.218945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.218980 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:50.219478 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.219489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.221222 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.221229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.224204 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.224215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.225990 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.225996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.226199 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.226231 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.226316 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.226434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18529,"mutex_wait_us":60,"thread_start_us":17936,"threads_started":1}
I20260619 01:02:50.226893 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.199634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.97.2:40235)
0619 01:02:50.216541 (+ 16907us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.216551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.217972 (+ 1421us) server_negotiation.cc:407] Connection header received
0619 01:02:50.218126 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.218133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.218360 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.218545 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:50.219684 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.221111 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.224364 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.225778 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.225886 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.226062 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.226402 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.226560 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16736,"thread_start_us":17391,"threads_started":1}
I20260619 01:02:50.226985 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.198794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45965 (local address 127.0.0.1:52012)
0619 01:02:50.217509 (+ 18715us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.217526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.217697 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.218637 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.218644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.218668 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:50.219450 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.219468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.221110 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.221118 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.223956 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.223969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.226126 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.226133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.226261 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.226283 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.226379 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.226499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18464,"mutex_wait_us":127,"thread_start_us":12496,"threads_started":1}
I20260619 01:02:50.227622 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.199478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52012 (local address 127.0.97.3:45965)
0619 01:02:50.217002 (+ 17524us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.217013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.217645 (+ 632us) server_negotiation.cc:407] Connection header received
0619 01:02:50.217881 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.217887 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.218158 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.218349 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:50.219652 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.220971 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.224133 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.225601 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.225746 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.225972 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.227163 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.227289 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17371,"thread_start_us":17237,"threads_started":1}
I20260619 01:02:50.231060 15169 tablet_service.cc:1467] Processing CreateTablet for tablet be4f641e2a17468aae748a5c91c92c0f (DEFAULT_TABLE table=test-table [id=fd7bd326fe564145884afbb5ecb7fbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:50.232527 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4f641e2a17468aae748a5c91c92c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:50.234246 15038 tablet_service.cc:1467] Processing CreateTablet for tablet be4f641e2a17468aae748a5c91c92c0f (DEFAULT_TABLE table=test-table [id=fd7bd326fe564145884afbb5ecb7fbef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:02:50.235131 15250 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Bootstrap starting.
I20260619 01:02:50.235594 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be4f641e2a17468aae748a5c91c92c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:02:50.238176 15250 tablet_bootstrap.cc:654] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:50.239194 15250 log.cc:824] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:50.242122 15250 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: No bootstrap required, opened a new log
I20260619 01:02:50.242369 15250 ts_tablet_manager.cc:1397] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 01:02:50.248147 15252 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Bootstrap starting.
I20260619 01:02:50.251013 15250 raft_consensus.cc:348] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.251366 15253 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Bootstrap starting.
I20260619 01:02:50.251547 15250 raft_consensus.cc:374] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:50.251636 15252 tablet_bootstrap.cc:654] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:50.251742 15250 raft_consensus.cc:729] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e588cbac6ec47a293bc594294122999, State: Initialized, Role: FOLLOWER
I20260619 01:02:50.252506 15250 consensus_queue.cc:260] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.252876 15252 log.cc:824] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:50.253649 14971 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:50.254405 15250 ts_tablet_manager.cc:1428] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 01:02:50.255424 15253 tablet_bootstrap.cc:654] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Neither blocks nor log segments found. Creating new log.
I20260619 01:02:50.256362 15253 log.cc:824] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:50.259097 15253 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: No bootstrap required, opened a new log
I20260619 01:02:50.259356 15253 ts_tablet_manager.cc:1397] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 01:02:50.261739 15252 tablet_bootstrap.cc:492] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: No bootstrap required, opened a new log
I20260619 01:02:50.261981 15252 ts_tablet_manager.cc:1397] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260619 01:02:50.267552 15253 raft_consensus.cc:348] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.267958 15253 raft_consensus.cc:374] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:50.268060 15253 raft_consensus.cc:729] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0e53dd363954afbae6c5b2b7a8585d2, State: Initialized, Role: FOLLOWER
W20260619 01:02:50.268328 14972 tablet.cc:2369] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:50.268846 15253 consensus_queue.cc:260] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.269009 15252 raft_consensus.cc:348] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.269601 15252 raft_consensus.cc:374] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:02:50.269745 15252 raft_consensus.cc:729] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6e5549b7bb494eba94ac967cd284fc, State: Initialized, Role: FOLLOWER
I20260619 01:02:50.270025 15103 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:50.270479 15253 ts_tablet_manager.cc:1428] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 01:02:50.270506 15252 consensus_queue.cc:260] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.271605 15234 heartbeater.cc:503] Master 127.0.97.62:39657 was elected leader, sending a full tablet report...
I20260619 01:02:50.272176 15252 ts_tablet_manager.cc:1428] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 01:02:50.309984 15257 raft_consensus.cc:482] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:02:50.310321 15257 raft_consensus.cc:504] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.311928 15257 leader_election.cc:288] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171), 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965)
I20260619 01:02:50.318820 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.312495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52485 (local address 127.0.97.3:45965)
0619 01:02:50.312715 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.312721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.313303 (+ 582us) server_negotiation.cc:407] Connection header received
0619 01:02:50.313481 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.313485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.313559 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.313660 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:50.314333 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.315142 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.317035 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.317819 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.317977 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.318075 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.318386 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.318522 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 01:02:50.318822 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.312196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41829 (local address 127.0.97.1:44171)
0619 01:02:50.312470 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.312479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.313144 (+ 665us) server_negotiation.cc:407] Connection header received
0619 01:02:50.313401 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.313408 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.313496 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.313562 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:02:50.314353 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.315152 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.317012 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.317775 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.317870 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.317986 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.318388 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.318526 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:02:50.319024 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.312090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.97.2:41829)
0619 01:02:50.313022 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.313042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.313218 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.313604 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.313609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.313638 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:50.314179 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.314191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.315295 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.315299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.316848 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.316860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.317999 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.318005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.318223 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.318252 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.318356 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.318542 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":55,"thread_start_us":649,"threads_started":1}
I20260619 01:02:50.319080 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.312333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45965 (local address 127.0.97.2:52485)
0619 01:02:50.313206 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.313221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:02:50.313357 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:02:50.313698 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:02:50.313704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:02:50.313740 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:02:50.314175 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.314185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.315268 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.315273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.316853 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:02:50.316865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.318064 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.318069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:02:50.318218 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.318249 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:02:50.318361 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 01:02:50.318545 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":137,"thread_start_us":521,"threads_started":1}
I20260619 01:02:50.319962 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4f641e2a17468aae748a5c91c92c0f" candidate_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e588cbac6ec47a293bc594294122999" is_pre_election: true
I20260619 01:02:50.320170 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4f641e2a17468aae748a5c91c92c0f" candidate_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6e5549b7bb494eba94ac967cd284fc" is_pre_election: true
I20260619 01:02:50.320472 14926 raft_consensus.cc:2454] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e53dd363954afbae6c5b2b7a8585d2 in term 0.
I20260619 01:02:50.320699 15189 raft_consensus.cc:2454] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0e53dd363954afbae6c5b2b7a8585d2 in term 0.
I20260619 01:02:50.321151 14993 leader_election.cc:302] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e588cbac6ec47a293bc594294122999, a0e53dd363954afbae6c5b2b7a8585d2; no voters:
I20260619 01:02:50.321547 15257 raft_consensus.cc:2781] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:02:50.321709 15257 raft_consensus.cc:482] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:02:50.321826 15257 raft_consensus.cc:3037] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:50.324193 15257 raft_consensus.cc:504] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.324844 15257 leader_election.cc:288] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [CANDIDATE]: Term 1 election: Requested vote from peers 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171), 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965)
I20260619 01:02:50.325248 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4f641e2a17468aae748a5c91c92c0f" candidate_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e588cbac6ec47a293bc594294122999"
I20260619 01:02:50.325299 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be4f641e2a17468aae748a5c91c92c0f" candidate_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b6e5549b7bb494eba94ac967cd284fc"
I20260619 01:02:50.325436 14926 raft_consensus.cc:3037] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:50.325506 15189 raft_consensus.cc:3037] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:02:50.327724 14926 raft_consensus.cc:2454] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e53dd363954afbae6c5b2b7a8585d2 in term 1.
I20260619 01:02:50.328085 15189 raft_consensus.cc:2454] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0e53dd363954afbae6c5b2b7a8585d2 in term 1.
I20260619 01:02:50.328217 14993 leader_election.cc:302] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e588cbac6ec47a293bc594294122999, a0e53dd363954afbae6c5b2b7a8585d2; no voters:
I20260619 01:02:50.328500 15257 raft_consensus.cc:2781] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:02:50.329022 15257 raft_consensus.cc:686] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 LEADER]: Becoming Leader. State: Replica: a0e53dd363954afbae6c5b2b7a8585d2, State: Running, Role: LEADER
I20260619 01:02:50.329474 15257 consensus_queue.cc:237] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } }
I20260619 01:02:50.334182 14785 catalog_manager.cc:5495] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0e53dd363954afbae6c5b2b7a8585d2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40235 } health_report { overall_health: HEALTHY } } }
I20260619 01:02:50.374434 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.369486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58398 (local address 127.0.97.2:40235)
0619 01:02:50.369683 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.369689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.369770 (+ 81us) server_negotiation.cc:407] Connection header received
0619 01:02:50.369955 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.369959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.370028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.370147 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:02:50.370492 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.371182 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.372611 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.372982 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.373064 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.373438 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:02:50.373962 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:02:50.374090 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.374172 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.374229 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:02:50.381930 14926 raft_consensus.cc:1261] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Refusing update from remote peer a0e53dd363954afbae6c5b2b7a8585d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:50.381963 15189 raft_consensus.cc:1261] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Refusing update from remote peer a0e53dd363954afbae6c5b2b7a8585d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:02:50.382889 15261 consensus_queue.cc:1035] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b6e5549b7bb494eba94ac967cd284fc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:02:50.383215 15257 consensus_queue.cc:1035] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e588cbac6ec47a293bc594294122999" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:02:50.387303 15235 tablet.cc:2369] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:02:50.393229 15264 mvcc.cc:205] Tried to move back new op lower bound from 7298379654675099648 to 7298379654473494528. Current Snapshot: MvccSnapshot[applied={T|T < 7298379654675099648}]
I20260619 01:02:50.394129 15266 mvcc.cc:205] Tried to move back new op lower bound from 7298379654675099648 to 7298379654473494528. Current Snapshot: MvccSnapshot[applied={T|T < 7298379654675099648}]
I20260619 01:02:50.406075 14784 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:50.406388 14784 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:02:50.410022 14784 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Sending DeleteTablet for 3 replicas of tablet be4f641e2a17468aae748a5c91c92c0f
I20260619 01:02:50.411065 14906 tablet_service.cc:1514] Processing DeleteTablet for tablet be4f641e2a17468aae748a5c91c92c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52672
I20260619 01:02:50.411234 15169 tablet_service.cc:1514] Processing DeleteTablet for tablet be4f641e2a17468aae748a5c91c92c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52012
I20260619 01:02:50.411444 15038 tablet_service.cc:1514] Processing DeleteTablet for tablet be4f641e2a17468aae748a5c91c92c0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:02:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41832
I20260619 01:02:50.412359 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14822
I20260619 01:02:50.422310 15275 tablet_replica.cc:330] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: stopping tablet replica
I20260619 01:02:50.422858 15275 raft_consensus.cc:2229] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:50.423388 15275 raft_consensus.cc:2258] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:50.423633 15274 tablet_replica.cc:330] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: stopping tablet replica
I20260619 01:02:50.424134 15276 tablet_replica.cc:330] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: stopping tablet replica
I20260619 01:02:50.424077 15274 raft_consensus.cc:2229] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:02:50.424499 15276 raft_consensus.cc:2229] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:02:50.424585 15274 raft_consensus.cc:2258] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:50.425135 15276 raft_consensus.cc:2258] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:02:50.426999 15274 ts_tablet_manager.cc:1905] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:50.427112 15276 ts_tablet_manager.cc:1905] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:50.429888 15275 ts_tablet_manager.cc:1905] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:50.433169 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.413736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.1:44171)
0619 01:02:50.413896 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.413902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.413918 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:02:50.413968 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.413973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.414047 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.414140 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:50.416083 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.417225 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.419622 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.420330 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.420450 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.424820 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:50.424834 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:50.424851 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:50.424890 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:50.429103 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.430035 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.430044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.430047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.430194 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.430668 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.430673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.430677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.431139 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:50.431445 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.432551 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.432794 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:02:50.433923 14966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:50.438788 15275 ts_tablet_manager.cc:1918] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:50.438863 15276 ts_tablet_manager.cc:1918] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:50.439006 15276 log.cc:1196] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/be4f641e2a17468aae748a5c91c92c0f
I20260619 01:02:50.439033 15275 log.cc:1196] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/be4f641e2a17468aae748a5c91c92c0f
I20260619 01:02:50.439566 15276 ts_tablet_manager.cc:1939] T be4f641e2a17468aae748a5c91c92c0f P a0e53dd363954afbae6c5b2b7a8585d2: Deleting consensus metadata
I20260619 01:02:50.439700 15275 ts_tablet_manager.cc:1939] T be4f641e2a17468aae748a5c91c92c0f P 3b6e5549b7bb494eba94ac967cd284fc: Deleting consensus metadata
I20260619 01:02:50.441294 14770 catalog_manager.cc:4841] TS 3b6e5549b7bb494eba94ac967cd284fc (127.0.97.3:45965): tablet be4f641e2a17468aae748a5c91c92c0f (table test-table [id=fd7bd326fe564145884afbb5ecb7fbef]) successfully deleted
I20260619 01:02:50.441359 14772 catalog_manager.cc:4841] TS a0e53dd363954afbae6c5b2b7a8585d2 (127.0.97.2:40235): tablet be4f641e2a17468aae748a5c91c92c0f (table test-table [id=fd7bd326fe564145884afbb5ecb7fbef]) successfully deleted
I20260619 01:02:50.529410 15274 ts_tablet_manager.cc:1918] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:50.529649 15274 log.cc:1196] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/be4f641e2a17468aae748a5c91c92c0f
I20260619 01:02:50.530120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14822
I20260619 01:02:50.530169 15274 ts_tablet_manager.cc:1939] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Deleting consensus metadata
W20260619 01:02:50.553781 14771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44171 (error 108)
I20260619 01:02:50.554075 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14975
I20260619 01:02:50.559021 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.558445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52010)
0619 01:02:50.558633 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.558821 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:02:50.559326 14771 catalog_manager.cc:4584] TS 4e588cbac6ec47a293bc594294122999 (127.0.97.1:44171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be4f641e2a17468aae748a5c91c92c0f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
I20260619 01:02:50.581274 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.555210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.0.97.2:40235)
0619 01:02:50.555458 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.555467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.562041 (+ 6574us) server_negotiation.cc:407] Connection header received
0619 01:02:50.562115 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.562122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.562206 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.562333 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:50.565429 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.566702 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.568704 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.569447 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.569596 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.573655 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:50.573670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:50.573688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:50.573746 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:50.577897 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.578841 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.578851 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.578855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.579048 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.579533 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.579543 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.579547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.579993 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:50.580311 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.580686 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.580898 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 01:02:50.582060 15098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:50.612812 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.612380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52020)
0619 01:02:50.612577 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.612669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 01:02:50.613909 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.613560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52036)
0619 01:02:50.613732 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.613806 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 01:02:50.674337 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.673899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52040)
0619 01:02:50.674090 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.674175 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 01:02:50.675345 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.675000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52046)
0619 01:02:50.675146 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.675224 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 01:02:50.693387 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14975
I20260619 01:02:50.712028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15106
I20260619 01:02:50.725585 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.712788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.0.97.3:45965)
0619 01:02:50.712994 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:02:50.713000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:02:50.713095 (+ 95us) server_negotiation.cc:407] Connection header received
0619 01:02:50.713301 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:02:50.713306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:02:50.713381 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:02:50.713487 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:02:50.714804 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.715509 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.716909 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:02:50.717322 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:02:50.717459 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:02:50.720190 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:02:50.720203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:02:50.720218 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:02:50.720252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:02:50.722821 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.723444 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.723456 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.723459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.723618 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:02:50.723982 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:02:50.723987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:02:50.723990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:02:50.724386 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:02:50.724684 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 01:02:50.725018 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 01:02:50.725242 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:02:50.726220 15229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:50.756138 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.755602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52062)
0619 01:02:50.755839 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.755949 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 01:02:50.757213 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.756772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52066)
0619 01:02:50.756981 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.757080 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260619 01:02:50.815428 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15106
I20260619 01:02:50.833007 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 14752
I20260619 01:02:50.834100 14815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:02:50.944921 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 14752
I20260619 01:02:50.945009 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:50.944197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44171 (local address 127.0.0.1:52072)
0619 01:02:50.944747 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:50.944854 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401}
I20260619 01:02:50.965242 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--webserver_interface=127.0.97.62
--webserver_port=44305
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:51.275563 15282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:51.275830 15282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:51.275978 15282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:51.282641 15282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:02:51.282801 15282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:51.282877 15282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:51.282974 15282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:02:51.283056 15282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:02:51.292724 15282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:51.293875 15282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:51.295178 15282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:51.306272 15287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:51.306288 15288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:51.306252 15290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:51.307245 15282 server_base.cc:1034] running on GCE node
I20260619 01:02:51.307915 15282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:51.309180 15282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:51.310379 15282 hybrid_clock.cc:648] HybridClock initialized: now 1781830971310329 us; error 48 us; skew 500 ppm
I20260619 01:02:51.310748 15282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:51 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:51.313746 15282 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:02:51.315222 15282 webserver.cc:460] Webserver started at http://127.0.97.62:44305/ using document root <none> and password file <none>
I20260619 01:02:51.315855 15282 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:51.315981 15282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:51.320981 15282 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 01:02:51.323580 15297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:51.324788 15282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 01:02:51.324965 15282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "897701f45a4c412f8cb2f4bb0900eb91"
format_stamp: "Formatted at 2026-06-19 01:02:45 on dist-test-slave-3rxd"
server_key: "ae54e33bd424bc6a86d74f8344b1c761"
server_key_iv: "ca398767a7d67ec3b582b58063c41b54"
server_key_version: "encryptionkey@0"
I20260619 01:02:51.325865 15282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:51.339813 15282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:51.343420 15282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:51.343732 15282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:51.361205 15282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39657
I20260619 01:02:51.361244 15348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39657 every 8 connection(s)
I20260619 01:02:51.362599 15282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:02:51.364555 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15282
I20260619 01:02:51.365518 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44171
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=44345
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:51.371843 15349 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:02:51.379762 15349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap starting.
I20260619 01:02:51.399145 15349 log.cc:824] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Log is configured to *not* fsync() on all Append() calls
I20260619 01:02:51.414513 15349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:02:51.415109 15349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91: Bootstrap complete.
I20260619 01:02:51.424043 15349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:51.424904 15349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Initialized, Role: FOLLOWER
I20260619 01:02:51.425472 15349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:51.425725 15349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:02:51.425863 15349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:02:51.426036 15349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:02:51.429724 15349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:51.430199 15349 leader_election.cc:302] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 897701f45a4c412f8cb2f4bb0900eb91; no voters:
I20260619 01:02:51.430898 15349 leader_election.cc:288] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 01:02:51.431049 15353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 01:02:51.432101 15353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [term 3 LEADER]: Becoming Leader. State: Replica: 897701f45a4c412f8cb2f4bb0900eb91, State: Running, Role: LEADER
I20260619 01:02:51.432880 15349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:02:51.432749 15353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } }
I20260619 01:02:51.435190 15354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:51.435500 15354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:51.439771 15355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 897701f45a4c412f8cb2f4bb0900eb91. Latest consensus state: current_term: 3 leader_uuid: "897701f45a4c412f8cb2f4bb0900eb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "897701f45a4c412f8cb2f4bb0900eb91" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39657 } } }
I20260619 01:02:51.440215 15355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 897701f45a4c412f8cb2f4bb0900eb91 [sys.catalog]: This master's current role is: LEADER
I20260619 01:02:51.441275 15358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:02:51.451236 15358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:02:51.453061 15358 catalog_manager.cc:1259] Loaded cluster ID: 26bc6081faf343629c9df14e4374f88a
I20260619 01:02:51.453187 15358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 01:02:51.455672 15358 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3c010e89ad google::LogMessage::Fail() at ??:0
@ 0x7f3c010eb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3c010e84de google::LogMessage::Flush() at ??:0
@ 0x7f3c010e9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3c16b1b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3c16b18976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3c16ba0610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3c16ba0439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3c03d34345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3c03d3791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3c03d37799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3c03d12eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3c06fec6db start_thread at ??:0
@ 0x7f3bfff8261f clone at ??:0
W20260619 01:02:51.758009 15351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:51.758296 15351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:51.758396 15351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:51.765942 15351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:51.766047 15351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:51.766162 15351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:02:51.776036 15351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=44345
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:51.777236 15351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:51.778468 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:51.791742 15389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:51.791812 15392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:51.791864 15390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:51.792797 15351 server_base.cc:1034] running on GCE node
I20260619 01:02:51.793375 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:51.794229 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:51.795425 15351 hybrid_clock.cc:648] HybridClock initialized: now 1781830971795354 us; error 67 us; skew 500 ppm
I20260619 01:02:51.795755 15351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:51 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:51.798844 15351 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:02:51.800259 15351 webserver.cc:460] Webserver started at http://127.0.97.1:44345/ using document root <none> and password file <none>
I20260619 01:02:51.800930 15351 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:51.801060 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:51.806797 15351 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:02:51.811353 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:51.812770 15351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:02:51.812943 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "4e588cbac6ec47a293bc594294122999"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "20fd384bc4853622d84222bd3ff47f95"
server_key_iv: "2803bd0a7cbfa6ccfba4cd2ce349a2b3"
server_key_version: "encryptionkey@0"
I20260619 01:02:51.813961 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:51.828480 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:51.832479 15351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:51.832762 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:51.833678 15351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:51.835325 15406 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 01:02:51.840322 15406 ts_tablet_manager.cc:1852] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 01:02:51.841254 15406 ts_tablet_manager.cc:1905] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:02:51.848120 15406 ts_tablet_manager.cc:1918] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:02:51.848388 15406 ts_tablet_manager.cc:1939] T be4f641e2a17468aae748a5c91c92c0f P 4e588cbac6ec47a293bc594294122999: Deleting consensus metadata
I20260619 01:02:51.848783 15351 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 01:02:51.848915 15351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.000s
I20260619 01:02:51.849061 15351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:51.849148 15351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:51.886346 15351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44171
I20260619 01:02:51.886412 15512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44171 every 8 connection(s)
I20260619 01:02:51.887933 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:02:51.892022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15351
I20260619 01:02:51.892585 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:51.890780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:45729)
0619 01:02:51.891507 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:51.891712 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":266,"threads_started":1}
I20260619 01:02:51.892597 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40235
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=45703
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:02:51.893275 15513 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39657 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39657: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
I20260619 01:02:51.894107 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:51.893809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:43413)
0619 01:02:51.893941 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:51.894003 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 01:02:51.895017 15515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:51.894699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:54245)
0619 01:02:51.894833 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:51.894901 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260619 01:02:51.895277 15513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:02:52.208385 15516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:52.208645 15516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:52.208842 15516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:52.215744 15516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:52.215860 15516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:52.216038 15516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:02:52.225394 15516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45703
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:52.226523 15516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:52.227737 15516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:52.240345 15521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:52.240348 15522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:52.240330 15524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:52.241206 15516 server_base.cc:1034] running on GCE node
I20260619 01:02:52.241802 15516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:52.242691 15516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:52.243940 15516 hybrid_clock.cc:648] HybridClock initialized: now 1781830972243890 us; error 47 us; skew 500 ppm
I20260619 01:02:52.244236 15516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:52 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:52.247164 15516 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:02:52.248494 15516 webserver.cc:460] Webserver started at http://127.0.97.2:45703/ using document root <none> and password file <none>
I20260619 01:02:52.249111 15516 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:52.249239 15516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:52.254204 15516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:02:52.256757 15531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.257939 15516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:02:52.258107 15516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "a0e53dd363954afbae6c5b2b7a8585d2"
format_stamp: "Formatted at 2026-06-19 01:02:46 on dist-test-slave-3rxd"
server_key: "1467366a6b679d5d09bf39fd438e5712"
server_key_iv: "42806196fa0d5519ed76ff05b3f9f7cb"
server_key_version: "encryptionkey@0"
I20260619 01:02:52.259120 15516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:52.291676 15516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:52.295378 15516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:52.295645 15516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:52.296528 15516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:52.297803 15516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:52.297921 15516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.298055 15516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:52.298143 15516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.330300 15516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40235
I20260619 01:02:52.330349 15643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40235 every 8 connection(s)
I20260619 01:02:52.331744 15516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:02:52.334074 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15516
I20260619 01:02:52.334755 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:45965
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39657
--webserver_port=35157
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:02:52.337177 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.334867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:41949)
0619 01:02:52.335757 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.336008 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":288,"threads_started":1}
W20260619 01:02:52.337845 15644 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39657 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39657: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
I20260619 01:02:52.338791 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.338354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:47169)
0619 01:02:52.338534 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.338617 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 01:02:52.339847 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.339428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:51627)
0619 01:02:52.339603 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.339698 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:02:52.340088 15644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:02:52.642952 15647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:02:52.643213 15647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:02:52.643404 15647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:02:52.650388 15647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:02:52.650483 15647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:02:52.650686 15647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:02:52.659977 15647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:45965
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=35157
--tserver_master_addrs=127.0.97.62:39657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:02:52.661201 15647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:02:52.662475 15647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:02:52.675629 15652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:52.675633 15653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:02:52.675724 15655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:02:52.676312 15647 server_base.cc:1034] running on GCE node
I20260619 01:02:52.676851 15647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:02:52.677757 15647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:02:52.678964 15647 hybrid_clock.cc:648] HybridClock initialized: now 1781830972678926 us; error 50 us; skew 500 ppm
I20260619 01:02:52.679256 15647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:02:52 dist-test-slave-3rxd krb5kdc[14200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:02:52.682142 15647 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:02:52.683475 15647 webserver.cc:460] Webserver started at http://127.0.97.3:35157/ using document root <none> and password file <none>
I20260619 01:02:52.684090 15647 fs_manager.cc:359] Metadata directory not provided
I20260619 01:02:52.684228 15647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:02:52.690212 15647 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 01:02:52.692992 15662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.694367 15647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:02:52.694542 15647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "3b6e5549b7bb494eba94ac967cd284fc"
format_stamp: "Formatted at 2026-06-19 01:02:47 on dist-test-slave-3rxd"
server_key: "853a84e74c465245af2b2976aef69f47"
server_key_iv: "da237d2556229fad108d1d43246b2422"
server_key_version: "encryptionkey@0"
I20260619 01:02:52.695574 15647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:02:52.712174 15647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:02:52.716045 15647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:02:52.716320 15647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:02:52.717394 15647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:02:52.718767 15647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:02:52.718895 15647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.719031 15647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:02:52.719120 15647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:02:52.751437 15647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45965
I20260619 01:02:52.751521 15774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45965 every 8 connection(s)
I20260619 01:02:52.752921 15647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:02:52.756374 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15647
I20260619 01:02:52.757741 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.755755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:43463)
0619 01:02:52.756607 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.756868 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":328,"threads_started":1}
W20260619 01:02:52.758436 15775 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39657 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:39657: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
I20260619 01:02:52.759286 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.758957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:44169)
0619 01:02:52.759112 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.759185 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 01:02:52.760124 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.759806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:35751)
0619 01:02:52.759928 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.760000 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:02:52.760385 15775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:02:52.896490 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:52.895764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:49879)
0619 01:02:52.896195 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:52.896275 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":220,"threads_started":1}
I20260619 01:02:53.341346 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:53.340573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:32911)
0619 01:02:53.341018 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:53.341116 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260619 01:02:53.761499 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:53.760821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:50249)
0619 01:02:53.761214 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:53.761291 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":186,"threads_started":1}
I20260619 01:02:53.898051 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:53.897300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:53939)
0619 01:02:53.897790 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:53.897868 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":305,"threads_started":1}
I20260619 01:02:54.342907 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:54.342178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:42525)
0619 01:02:54.342595 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:54.342683 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260619 01:02:54.763060 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:54.762293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:48575)
0619 01:02:54.762817 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:54.762903 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":212,"threads_started":1}
I20260619 01:02:54.899847 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:54.899086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:46561)
0619 01:02:54.899543 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:54.899642 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":229,"threads_started":1}
I20260619 01:02:55.344261 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:55.343652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:59783)
0619 01:02:55.344037 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:55.344111 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":200,"threads_started":1}
I20260619 01:02:55.764645 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:55.763909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:46361)
0619 01:02:55.764409 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:55.764484 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":289,"threads_started":1}
I20260619 01:02:55.901458 15788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:55.900781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:39527)
0619 01:02:55.901218 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:55.901293 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":232,"threads_started":1}
I20260619 01:02:56.345846 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:56.345167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:35965)
0619 01:02:56.345597 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:56.345686 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":233,"threads_started":1}
I20260619 01:02:56.766127 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:56.765439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:57101)
0619 01:02:56.765858 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:56.765936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":201,"threads_started":1}
I20260619 01:02:56.903018 15791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:56.902318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:48161)
0619 01:02:56.902795 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:56.902871 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":199,"threads_started":1}
I20260619 01:02:57.347368 15792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:57.346672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:41733)
0619 01:02:57.347140 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:57.347211 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":291,"threads_started":1}
I20260619 01:02:57.767719 15793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:57.767072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:46399)
0619 01:02:57.767482 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:57.767555 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":212,"threads_started":1}
I20260619 01:02:57.904518 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:57.903867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:47897)
0619 01:02:57.904277 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:57.904352 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":211,"threads_started":1}
I20260619 01:02:58.348892 15795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:58.348210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:35063)
0619 01:02:58.348647 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:58.348732 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260619 01:02:58.769363 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:58.768666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:40865)
0619 01:02:58.769107 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:58.769204 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":237,"threads_started":1}
I20260619 01:02:58.906193 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:58.905438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:33751)
0619 01:02:58.905917 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:58.905992 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":223,"threads_started":1}
I20260619 01:02:59.350589 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:59.349804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:48931)
0619 01:02:59.350300 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:59.350384 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":200,"threads_started":1}
I20260619 01:02:59.771037 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:59.770206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:39979)
0619 01:02:59.770691 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:59.770827 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":235,"threads_started":1}
I20260619 01:02:59.907887 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:02:59.907205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:48729)
0619 01:02:59.907639 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 01:02:59.907715 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":239,"threads_started":1}
I20260619 01:03:00.352052 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:00.351425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:50595)
0619 01:03:00.351824 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:00.351900 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":209,"threads_started":1}
I20260619 01:03:00.772580 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:00.771928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:45195)
0619 01:03:00.772347 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:00.772426 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260619 01:03:00.909823 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:00.908783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:51747)
0619 01:03:00.909470 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:00.909590 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":393,"threads_started":1}
I20260619 01:03:01.353596 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:01.352920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:58709)
0619 01:03:01.353341 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:01.353429 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":211,"threads_started":1}
I20260619 01:03:01.774312 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:01.773480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:53421)
0619 01:03:01.773954 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:01.774078 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":253,"threads_started":1}
I20260619 01:03:01.911384 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:01.910697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:50155)
0619 01:03:01.911134 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:01.911210 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":188,"threads_started":1}
I20260619 01:03:02.355348 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:02.354593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:34201)
0619 01:03:02.355094 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:02.355174 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":307,"threads_started":1}
I20260619 01:03:02.775913 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:02.775240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:38697)
0619 01:03:02.775685 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:02.775761 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":218,"threads_started":1}
I20260619 01:03:02.912943 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:02.912249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:54779)
0619 01:03:02.912703 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:02.912781 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":229,"threads_started":1}
I20260619 01:03:03.356886 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:03.356211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:41677)
0619 01:03:03.356619 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:03.356696 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":217,"threads_started":1}
I20260619 01:03:03.777472 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:03.776787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:53369)
0619 01:03:03.777212 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:03.777299 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":219,"threads_started":1}
I20260619 01:03:03.914572 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:03.913891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:55967)
0619 01:03:03.914318 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:03.914393 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":231,"threads_started":1}
I20260619 01:03:04.358392 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:04.357697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:42925)
0619 01:03:04.358125 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:04.358201 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":238,"threads_started":1}
I20260619 01:03:04.779130 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:04.778412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:47547)
0619 01:03:04.778893 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:04.778977 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":225,"threads_started":1}
I20260619 01:03:04.916152 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:04.915448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:41105)
0619 01:03:04.915897 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:04.915975 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260619 01:03:05.359961 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:05.359289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:39153)
0619 01:03:05.359698 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:05.359778 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":204,"threads_started":1}
I20260619 01:03:05.780633 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:05.779978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:39097)
0619 01:03:05.780375 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:05.780454 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":186,"threads_started":1}
I20260619 01:03:05.917821 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:05.917069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:50767)
0619 01:03:05.917532 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:05.917615 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":263,"threads_started":1}
I20260619 01:03:06.361572 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:06.360798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:56983)
0619 01:03:06.361304 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:06.361395 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":211,"threads_started":1}
I20260619 01:03:06.782295 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:06.781521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:53777)
0619 01:03:06.782000 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:06.782104 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260619 01:03:06.919526 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:06.918844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.1:33167)
0619 01:03:06.919290 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:06.919365 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260619 01:03:07.363569 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:07.362691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.2:60315)
0619 01:03:07.363276 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:07.363382 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":273,"threads_started":1}
I20260619 01:03:07.764103 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15351
I20260619 01:03:07.784339 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:07.783501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39657 (local address 127.0.97.3:38617)
0619 01:03:07.784013 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:07.784121 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260619 01:03:07.786455 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:07.765492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42934 (local address 127.0.97.1:44171)
0619 01:03:07.766268 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:07.766282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:07.766318 (+ 36us) server_negotiation.cc:407] Connection header received
0619 01:03:07.768660 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:07.768671 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:07.769107 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:07.769480 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:07.771083 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:07.772318 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:07.773571 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:07.774294 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:07.774498 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:07.780404 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:07.780412 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:07.780429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:07.780468 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:07.783277 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:07.784096 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:07.784103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:07.784110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:07.784316 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:07.784745 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:07.784749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:07.784751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:07.785283 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:07.785508 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:07.785835 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 01:03:07.786175 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":500,"threads_started":1}
I20260619 01:03:07.787914 15508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:07.871385 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15351
I20260619 01:03:07.888463 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15516
I20260619 01:03:07.905387 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:07.889435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.0.97.2:40235)
0619 01:03:07.889987 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:07.890000 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:07.890039 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:03:07.891634 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:07.891644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:07.892081 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:07.892396 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:07.893619 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:07.894920 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:07.896101 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:07.896749 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:07.896876 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:07.899789 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:07.899803 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:07.899823 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:07.899871 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:07.902490 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:07.903211 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:07.903218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:07.903223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:07.903393 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:07.903725 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:07.903728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:07.903730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:07.904184 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:07.904411 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:07.904810 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 01:03:07.905111 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":280,"threads_started":1}
I20260619 01:03:07.906869 15639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:07.988801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15516
I20260619 01:03:08.006462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15647
I20260619 01:03:08.023181 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:08.007500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.97.3:45965)
0619 01:03:08.008131 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:08.008145 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:08.008185 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:03:08.009675 (+ 1490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:08.009683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:08.010088 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:08.010419 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:08.011896 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:08.013018 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:08.014285 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:08.014982 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:08.015107 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:08.017552 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:08.017559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:08.017572 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:08.017604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:08.020310 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:08.020962 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:08.020967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:08.020973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:08.021126 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:08.021480 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:08.021483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:08.021484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:08.021933 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:08.022135 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:08.022515 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 01:03:08.022897 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":323,"threads_started":1}
I20260619 01:03:08.024377 15770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:08.105794 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15647
2026-06-19T01:03:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24206 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:03:08 dist-test-slave-3rxd krb5kdc[15832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:03:08 dist-test-slave-3rxd krb5kdc[15832](info): set up 2 sockets
Jun 19 01:03:08 dist-test-slave-3rxd krb5kdc[15832](info): commencing operation
krb5kdc: starting...
W20260619 01:03:09.191253 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:03:09 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:03:09.414186 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:09.731757 15845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:09.732092 15845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:09.732265 15845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:09.739338 15845 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:03:09.739482 15845 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:03:09.739552 15845 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:03:09.739640 15845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:03:09.739722 15845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:09.739816 15845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:09.739903 15845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:03:09.739980 15845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:03:09.750828 15845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:09.752102 15845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:09.753391 15845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:09.764871 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:09.764830 15853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:09.764940 15851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:09.765501 15845 server_base.cc:1034] running on GCE node
I20260619 01:03:09.766230 15845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:09.767606 15845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:09.768914 15845 hybrid_clock.cc:648] HybridClock initialized: now 1781830989768812 us; error 49 us; skew 500 ppm
I20260619 01:03:09.769275 15845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:09 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:09.772511 15845 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:03:09.774032 15845 webserver.cc:460] Webserver started at http://127.0.97.62:43913/ using document root <none> and password file <none>
I20260619 01:03:09.774755 15845 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:09.774890 15845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:09.775213 15845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:03:09.778007 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "cbfdffe046fc47e9bc903b2f59a56d06"
format_stamp: "Formatted at 2026-06-19 01:03:09 on dist-test-slave-3rxd"
server_key: "d8c5b50921d82ef9374fccf2dc23b75c"
server_key_iv: "20326c005bf8fa18c2229e9534fa180e"
server_key_version: "encryptionkey@0"
I20260619 01:03:09.779023 15845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "cbfdffe046fc47e9bc903b2f59a56d06"
format_stamp: "Formatted at 2026-06-19 01:03:09 on dist-test-slave-3rxd"
server_key: "d8c5b50921d82ef9374fccf2dc23b75c"
server_key_iv: "20326c005bf8fa18c2229e9534fa180e"
server_key_version: "encryptionkey@0"
I20260619 01:03:09.784548 15845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:03:09.787804 15860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:09.789028 15845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:03:09.789201 15845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "cbfdffe046fc47e9bc903b2f59a56d06"
format_stamp: "Formatted at 2026-06-19 01:03:09 on dist-test-slave-3rxd"
server_key: "d8c5b50921d82ef9374fccf2dc23b75c"
server_key_iv: "20326c005bf8fa18c2229e9534fa180e"
server_key_version: "encryptionkey@0"
I20260619 01:03:09.789407 15845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:09.805994 15845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:09.809664 15845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:09.809908 15845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:09.827855 15845 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35037
I20260619 01:03:09.827874 15911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35037 every 8 connection(s)
I20260619 01:03:09.829107 15845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:03:09.832942 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:09.834757 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15845
I20260619 01:03:09.834990 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:03:09.835777 388 external_mini_cluster.cc:1428] Setting key f2ef9f230bf204d31d65e6d8f6099d76
I20260619 01:03:09.845397 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap starting.
I20260619 01:03:09.849148 15912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:09.850337 15912 log.cc:824] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:03:09 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:09.853011 15912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: No bootstrap required, opened a new log
I20260619 01:03:09.859459 15915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:09.837765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.0.97.62:35037)
0619 01:03:09.838546 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:09.838562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:09.838604 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:03:09.840250 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:09.840270 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:09.840623 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:09.840930 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:09.842622 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:09.843998 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:09.845908 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:09.846581 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:09.846794 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:09.851510 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:09.851518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:09.851531 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:09.851569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:09.855248 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:09.855891 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:09.855901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:09.855913 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:09.856117 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:09.856458 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:09.856463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:09.856467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:09.857136 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:09.857457 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:09.857726 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:03:09.858128 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":383,"threads_started":1}
I20260619 01:03:09.862552 15912 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:09.862946 15912 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:09.863021 15912 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Initialized, Role: FOLLOWER
I20260619 01:03:09.863536 15912 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:09.863747 15912 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:03:09.863842 15912 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:03:09.863950 15912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:03:09.865921 15912 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:09.866405 15912 leader_election.cc:302] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfdffe046fc47e9bc903b2f59a56d06; no voters:
I20260619 01:03:09.866946 15912 leader_election.cc:288] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:03:09.867091 15917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:03:09.867766 15917 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 LEADER]: Becoming Leader. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Running, Role: LEADER
I20260619 01:03:09.868278 15917 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:09.868963 15912 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:03:09.871816 15919 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfdffe046fc47e9bc903b2f59a56d06. Latest consensus state: current_term: 1 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:09.872277 15919 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:09.872406 15918 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:09.872768 15918 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:09.873976 15926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:03:09.878176 15926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:03:09.885880 15926 catalog_manager.cc:1347] Generated new cluster ID: 076bfdc213964ce8b7c5b92aed3f28a1
I20260619 01:03:09.885989 15926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:03:09.896281 15926 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:03:09.897426 15926 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:03:09.907353 15926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Generated new TSK 0
I20260619 01:03:09.908097 15926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:03:09.960476 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:10.280377 15938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:10.280616 15938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:10.280702 15938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:10.287595 15938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:10.287699 15938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:10.287901 15938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:03:10.297298 15938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:10.298501 15938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:10.299823 15938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:10.312538 15944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:10.312574 15943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:10.312608 15946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:10.313256 15938 server_base.cc:1034] running on GCE node
I20260619 01:03:10.313860 15938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:10.314670 15938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:10.315912 15938 hybrid_clock.cc:648] HybridClock initialized: now 1781830990315869 us; error 34 us; skew 500 ppm
I20260619 01:03:10.316215 15938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:10 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:10.319242 15938 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:03:10.320515 15938 webserver.cc:460] Webserver started at http://127.0.97.1:36043/ using document root <none> and password file <none>
I20260619 01:03:10.321131 15938 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:10.321286 15938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:10.321576 15938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:03:10.324376 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "880a6e682bc00c1a8bb361652e2ed33b"
server_key_iv: "df18da0cf8786cc33b983486e1623a11"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.325091 15938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "880a6e682bc00c1a8bb361652e2ed33b"
server_key_iv: "df18da0cf8786cc33b983486e1623a11"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.329757 15938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 01:03:10.332985 15953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.334160 15938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:03:10.334348 15938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "880a6e682bc00c1a8bb361652e2ed33b"
server_key_iv: "df18da0cf8786cc33b983486e1623a11"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.334543 15938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:10.352878 15938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:10.356870 15938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:10.357157 15938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:10.358072 15938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:10.359426 15938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:10.359498 15938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.359575 15938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:10.359617 15938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.391182 15938 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35395
I20260619 01:03:10.391227 16065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35395 every 8 connection(s)
I20260619 01:03:10.392370 15938 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:03:10.393709 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 15938
I20260619 01:03:10.393936 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:03:10.394699 388 external_mini_cluster.cc:1428] Setting key a220444201ea2630a1994b4f0404f911
Jun 19 01:03:10 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:10.412241 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:10.395000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56577 (local address 127.0.97.62:35037)
0619 01:03:10.395684 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:10.395690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:10.396544 (+ 854us) server_negotiation.cc:407] Connection header received
0619 01:03:10.398394 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:10.398399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:10.398487 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:10.398614 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:10.400121 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.400825 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.402314 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.402694 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.402841 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:10.405965 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:10.405970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:10.405975 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:10.406013 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:10.409194 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:10.409890 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:10.409898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:10.409903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:10.410029 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:10.410511 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:10.410517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:10.410520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:10.410878 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:10.411032 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:10.411681 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 01:03:10.411898 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":459,"threads_started":1}
I20260619 01:03:10.413187 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:10.395660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:56577)
0619 01:03:10.396401 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:10.396441 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:10.398129 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:10.398873 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:10.398881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:10.399380 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:10.399949 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:10.399979 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.400973 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.400982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:10.402181 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:10.402191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.402923 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.402932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:10.403182 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:10.403841 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:10.403865 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:10.405779 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:10.409376 (+ 3597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:10.409382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:10.409393 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:10.409770 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:10.410175 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:10.410180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:10.410184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:10.410398 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:10.411002 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:10.411009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:10.411327 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 01:03:10.411598 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 01:03:10.412080 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260619 01:03:10.414829 16066 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:10.415128 16066 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:10.415854 16066 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260619 01:03:10.417910 15877 ts_manager.cc:194] Registered new tserver with Master: c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260619 01:03:10.419414 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:03:10.443334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:10.766800 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:10.767123 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:10.767220 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:10.774809 16072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:10.774909 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:10.775022 16072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:03:10.784670 16072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:10.785916 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:10.787314 16072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:10.800618 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:10.800645 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:10.800559 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:10.801292 16072 server_base.cc:1034] running on GCE node
I20260619 01:03:10.801852 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:10.802839 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:10.804075 16072 hybrid_clock.cc:648] HybridClock initialized: now 1781830990804033 us; error 52 us; skew 500 ppm
I20260619 01:03:10.804385 16072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:10 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:10.807372 16072 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:03:10.808821 16072 webserver.cc:460] Webserver started at http://127.0.97.2:34535/ using document root <none> and password file <none>
I20260619 01:03:10.809463 16072 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:10.809595 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:10.809931 16072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:03:10.812727 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "2d586709e3ff0aaa00c8d197b476d266"
server_key_iv: "4e2e21106ac1a6850aa55e80b1939e1a"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.813427 16072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "2d586709e3ff0aaa00c8d197b476d266"
server_key_iv: "4e2e21106ac1a6850aa55e80b1939e1a"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.818316 16072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:03:10.821805 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.823073 16072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:03:10.823242 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "2d586709e3ff0aaa00c8d197b476d266"
server_key_iv: "4e2e21106ac1a6850aa55e80b1939e1a"
server_key_version: "encryptionkey@0"
I20260619 01:03:10.823433 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:10.854331 16072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:10.858147 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:10.858439 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:10.859418 16072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:10.860634 16072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:10.860756 16072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.860893 16072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:10.860968 16072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:10.892872 16072 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44651
I20260619 01:03:10.892941 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44651 every 8 connection(s)
I20260619 01:03:10.894233 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:03:10.902987 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16072
I20260619 01:03:10.903172 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:03:10.903695 388 external_mini_cluster.cc:1428] Setting key 07724d23c9d520802ae2fbbd9e5cf84c
Jun 19 01:03:10 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:10.913200 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:10.896634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34957 (local address 127.0.97.62:35037)
0619 01:03:10.896850 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:10.896856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:10.898003 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 01:03:10.899782 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:10.899786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:10.899852 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:10.899956 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:10.901300 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.902014 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.903969 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.904329 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.904472 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:10.907829 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:10.907836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:10.907843 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:10.907895 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:10.910319 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:10.911110 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:10.911115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:10.911125 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:10.911243 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:10.911661 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:10.911665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:10.911668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:10.911918 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:10.912062 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:10.912678 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 01:03:10.912886 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 01:03:10.914021 16202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:10.897157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:34957)
0619 01:03:10.897840 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:10.897877 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:10.899522 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:10.900150 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:10.900157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:10.900630 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:10.901153 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:10.901170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.902137 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.902140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:10.903838 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:10.903853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:10.904559 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:10.904567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:10.904802 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:10.905484 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:10.905516 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:10.907528 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:10.910469 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:10.910477 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:10.910492 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:10.910988 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:10.911352 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:10.911358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:10.911362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:10.911546 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:10.912047 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:10.912054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:10.912349 (+ 295us) client_negotiation.cc:769] Sending connection context
0619 01:03:10.912584 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 01:03:10.913027 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":270,"threads_started":1}
I20260619 01:03:10.915339 16200 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:10.915640 16200 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:10.916437 16200 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260619 01:03:10.917901 15877 ts_manager.cc:194] Registered new tserver with Master: 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651)
I20260619 01:03:10.919186 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34957
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:03:10.954994 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:11.275621 16205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:11.275884 16205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:11.275977 16205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:11.282847 16205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:11.282950 16205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:11.283068 16205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:03:11.292335 16205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:11.293525 16205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:11.294884 16205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:11.307729 16211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:11.307727 16213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:11.307760 16210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:11.308900 16205 server_base.cc:1034] running on GCE node
I20260619 01:03:11.309473 16205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:11.310272 16205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:11.311519 16205 hybrid_clock.cc:648] HybridClock initialized: now 1781830991311490 us; error 41 us; skew 500 ppm
I20260619 01:03:11.311836 16205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:11 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:11.314847 16205 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:03:11.316159 16205 webserver.cc:460] Webserver started at http://127.0.97.3:43325/ using document root <none> and password file <none>
I20260619 01:03:11.316751 16205 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:11.316846 16205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:11.317088 16205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:03:11.319756 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "a7728afd14434b77bd7881f41d26ba3f"
format_stamp: "Formatted at 2026-06-19 01:03:11 on dist-test-slave-3rxd"
server_key: "e35f9acfc538b2c62f166a3fa17ee02e"
server_key_iv: "25cdb321cbf632158a0984fdd4b2cfe3"
server_key_version: "encryptionkey@0"
I20260619 01:03:11.320428 16205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a7728afd14434b77bd7881f41d26ba3f"
format_stamp: "Formatted at 2026-06-19 01:03:11 on dist-test-slave-3rxd"
server_key: "e35f9acfc538b2c62f166a3fa17ee02e"
server_key_iv: "25cdb321cbf632158a0984fdd4b2cfe3"
server_key_version: "encryptionkey@0"
I20260619 01:03:11.325078 16205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:03:11.328293 16220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:11.329478 16205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:03:11.329644 16205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "a7728afd14434b77bd7881f41d26ba3f"
format_stamp: "Formatted at 2026-06-19 01:03:11 on dist-test-slave-3rxd"
server_key: "e35f9acfc538b2c62f166a3fa17ee02e"
server_key_iv: "25cdb321cbf632158a0984fdd4b2cfe3"
server_key_version: "encryptionkey@0"
I20260619 01:03:11.329829 16205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:11.356447 16205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:11.360513 16205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:11.360793 16205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:11.361709 16205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:11.363013 16205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:11.363113 16205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:11.363256 16205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:11.363324 16205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:11.396193 16205 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37485
I20260619 01:03:11.396232 16332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37485 every 8 connection(s)
I20260619 01:03:11.397323 16205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:03:11.404461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16205
I20260619 01:03:11.404664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:03:11.405149 388 external_mini_cluster.cc:1428] Setting key c975b0e5ef1298ec053c40158b54ca04
Jun 19 01:03:11 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:11.416651 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.399750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34951 (local address 127.0.97.62:35037)
0619 01:03:11.399946 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.399951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.400988 (+ 1037us) server_negotiation.cc:407] Connection header received
0619 01:03:11.402829 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.402835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.402923 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.403036 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:11.404431 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.405398 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.407358 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.407877 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.408002 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.411239 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:11.411244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:11.411248 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:11.411280 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:11.413854 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.414451 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.414454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.414456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.414575 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.415038 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.415043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.415047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.415302 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:11.415468 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.416117 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.416336 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:03:11.417570 16335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.400198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:34951)
0619 01:03:11.400835 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.400874 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.402517 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.403262 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.403270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.403757 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:11.404270 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.404285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.405543 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.405549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.407199 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.407210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.408093 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.408100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.408349 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.408915 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:11.408937 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:11.411048 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:11.413960 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:11.413965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:11.413975 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:11.414350 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:11.414659 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:11.414661 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:11.414663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:11.414910 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:11.415437 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:11.415444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:11.415754 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.416002 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.416471 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":258,"threads_started":1}
I20260619 01:03:11.418926 16333 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:11.419236 16333 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:11.419929 16333 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260619 01:03:11.421267 15877 ts_manager.cc:194] Registered new tserver with Master: a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485)
I20260619 01:03:11.422022 15877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34951
I20260619 01:03:11.422081 16066 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260619 01:03:11.431980 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:03:11.448371 16068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.436289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.0.97.62:35037)
0619 01:03:11.436436 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.436440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.436953 (+ 513us) server_negotiation.cc:407] Connection header received
0619 01:03:11.437117 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.437120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.437183 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.437256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:11.438748 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.439537 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.440784 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.441185 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.441315 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.443590 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:11.443596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:11.443602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:11.443636 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:11.445777 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.446438 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.446443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.446447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.446572 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.446983 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.446990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.446993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.447274 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:11.447511 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.447848 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.448066 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:03:11.451896 15877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:03:11.458473 15877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:03:11.490235 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.470658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37485 (local address 127.0.0.1:46956)
0619 01:03:11.477627 (+ 6969us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.477670 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.477842 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.481957 (+ 4115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.481964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.481996 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:11.482454 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.482475 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.485299 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.485307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.487312 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.487333 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.489253 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.489261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.489465 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.489521 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.489654 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.489841 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6727,"thread_start_us":7287,"threads_started":1}
I20260619 01:03:11.490340 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.471699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.97.3:37485)
0619 01:03:11.481434 (+ 9735us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.481440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.481467 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:03:11.481557 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.481562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.481736 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.481950 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:11.482832 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.485072 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.487515 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.489019 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.489186 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.489419 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.489731 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.489911 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9589,"thread_start_us":9447,"threads_started":1}
I20260619 01:03:11.494593 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.471629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35395 (local address 127.0.0.1:57896)
0619 01:03:11.485691 (+ 14062us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.485716 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.485842 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.487369 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.487377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.487411 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:11.487801 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.487814 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.489792 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.489797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.492141 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.492153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.493828 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.493834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.493994 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.494026 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.494168 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.494280 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13894,"thread_start_us":14276,"threads_started":1}
I20260619 01:03:11.494808 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.472484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57896 (local address 127.0.97.1:35395)
0619 01:03:11.486636 (+ 14152us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.486648 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.486692 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:03:11.486844 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.486853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.487145 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.487374 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:11.488028 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.489598 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.492386 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.493636 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.493780 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.494005 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.494239 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.494414 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13938,"thread_start_us":15092,"threads_started":1}
I20260619 01:03:11.495832 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.471063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44651 (local address 127.0.0.1:59774)
0619 01:03:11.478287 (+ 7224us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.478311 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.478445 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.488746 (+ 10301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.488751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.488775 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:11.489155 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.489166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.491121 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.491128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.493172 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.493187 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.495317 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.495323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.495440 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.495463 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.495520 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.495611 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7049,"thread_start_us":6174,"threads_started":1}
I20260619 01:03:11.494521 16268 tablet_service.cc:1467] Processing CreateTablet for tablet 11b29a20dba64387847d68decb5791d3 (DEFAULT_TABLE table=test-table [id=81aaf59b2d0646d58f5995cec0302f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:11.497012 16268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b29a20dba64387847d68decb5791d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:11.497040 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.471153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.0.97.2:44651)
0619 01:03:11.488158 (+ 17005us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.488166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.488192 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:03:11.488285 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.488293 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.488505 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.488762 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:11.489353 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.490988 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.493341 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.495061 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.496219 (+ 1158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.496412 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.496490 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.496658 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16846,"thread_start_us":17287,"threads_started":1}
I20260619 01:03:11.498464 16001 tablet_service.cc:1467] Processing CreateTablet for tablet 11b29a20dba64387847d68decb5791d3 (DEFAULT_TABLE table=test-table [id=81aaf59b2d0646d58f5995cec0302f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:11.499691 16001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b29a20dba64387847d68decb5791d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:11.500396 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 11b29a20dba64387847d68decb5791d3 (DEFAULT_TABLE table=test-table [id=81aaf59b2d0646d58f5995cec0302f38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:11.502405 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11b29a20dba64387847d68decb5791d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:11.517824 16351 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Bootstrap starting.
I20260619 01:03:11.521801 16351 tablet_bootstrap.cc:654] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:11.522939 16352 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Bootstrap starting.
I20260619 01:03:11.523703 16353 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Bootstrap starting.
I20260619 01:03:11.523877 16351 log.cc:824] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:11.526679 16353 tablet_bootstrap.cc:654] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:11.527197 16352 tablet_bootstrap.cc:654] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:11.527943 16353 log.cc:824] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:11.528424 16352 log.cc:824] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:11.528882 16351 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: No bootstrap required, opened a new log
I20260619 01:03:11.529207 16351 ts_tablet_manager.cc:1397] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260619 01:03:11.530966 16352 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: No bootstrap required, opened a new log
I20260619 01:03:11.531090 16353 tablet_bootstrap.cc:492] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: No bootstrap required, opened a new log
I20260619 01:03:11.531245 16352 ts_tablet_manager.cc:1397] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 01:03:11.531353 16353 ts_tablet_manager.cc:1397] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 01:03:11.538924 16352 raft_consensus.cc:348] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.539292 16352 raft_consensus.cc:374] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:11.539391 16352 raft_consensus.cc:729] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7728afd14434b77bd7881f41d26ba3f, State: Initialized, Role: FOLLOWER
I20260619 01:03:11.539971 16352 consensus_queue.cc:260] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.540230 16351 raft_consensus.cc:348] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.540655 16351 raft_consensus.cc:374] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:11.540752 16351 raft_consensus.cc:729] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b86f8aad844b59abe8c8349ada0cc7, State: Initialized, Role: FOLLOWER
I20260619 01:03:11.541324 16351 consensus_queue.cc:260] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.542235 16353 raft_consensus.cc:348] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.542814 16353 raft_consensus.cc:374] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:11.542936 16353 raft_consensus.cc:729] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acd72c3e01e4959bcd0dcf0693bbce4, State: Initialized, Role: FOLLOWER
I20260619 01:03:11.543597 16353 consensus_queue.cc:260] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.546702 16351 ts_tablet_manager.cc:1428] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20260619 01:03:11.547557 16200 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260619 01:03:11.549269 16333 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260619 01:03:11.549633 16352 ts_tablet_manager.cc:1428] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Time spent starting tablet: real 0.018s user 0.007s sys 0.011s
I20260619 01:03:11.551167 16353 ts_tablet_manager.cc:1428] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20260619 01:03:11.630695 16358 raft_consensus.cc:482] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:03:11.630976 16358 raft_consensus.cc:504] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.632627 16358 leader_election.cc:288] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485), 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651)
I20260619 01:03:11.639544 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.633018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58835 (local address 127.0.97.3:37485)
0619 01:03:11.633237 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.633245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.633706 (+ 461us) server_negotiation.cc:407] Connection header received
0619 01:03:11.634003 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.634008 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.634127 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.634217 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:11.634852 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.635573 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.637518 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.638493 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.638586 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.638696 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.639153 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.639283 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:03:11.639694 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.633042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44651 (local address 127.0.97.1:44895)
0619 01:03:11.634039 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.634055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.634245 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.634744 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.634748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.634773 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:11.635039 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.635046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.636182 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.636189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.637615 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.637628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.638856 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.638861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.639055 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.639098 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.639201 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.639314 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"mutex_wait_us":2,"thread_start_us":668,"threads_started":1}
I20260619 01:03:11.639598 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.633429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44895 (local address 127.0.97.2:44651)
0619 01:03:11.633606 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.633614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.634215 (+ 601us) server_negotiation.cc:407] Connection header received
0619 01:03:11.634401 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.634407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.634485 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.634541 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:11.635181 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.636036 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.637757 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.638574 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.638757 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.638844 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.639238 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.639370 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 01:03:11.639971 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.632843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37485 (local address 127.0.97.1:58835)
0619 01:03:11.633603 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:11.633621 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:11.633837 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:11.634304 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:11.634311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:11.634340 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:11.634733 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.634744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.635699 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.635704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.637365 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:11.637377 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.638767 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.638774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:11.638995 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.639027 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:03:11.639104 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 01:03:11.639231 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":42,"thread_start_us":433,"threads_started":1}
I20260619 01:03:11.640671 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b29a20dba64387847d68decb5791d3" candidate_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" is_pre_election: true
I20260619 01:03:11.640980 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b29a20dba64387847d68decb5791d3" candidate_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7728afd14434b77bd7881f41d26ba3f" is_pre_election: true
I20260619 01:03:11.641151 16155 raft_consensus.cc:2454] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b86f8aad844b59abe8c8349ada0cc7 in term 0.
I20260619 01:03:11.641481 16288 raft_consensus.cc:2454] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b86f8aad844b59abe8c8349ada0cc7 in term 0.
I20260619 01:03:11.641839 15954 leader_election.cc:302] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8acd72c3e01e4959bcd0dcf0693bbce4, c6b86f8aad844b59abe8c8349ada0cc7; no voters:
I20260619 01:03:11.642256 16358 raft_consensus.cc:2781] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:03:11.642391 16358 raft_consensus.cc:482] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:03:11.642504 16358 raft_consensus.cc:3037] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Advancing to term 1
W20260619 01:03:11.644620 16067 tablet.cc:2369] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:03:11.644749 16358 raft_consensus.cc:504] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.645440 16358 leader_election.cc:288] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [CANDIDATE]: Term 1 election: Requested vote from peers a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485), 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651)
I20260619 01:03:11.645762 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b29a20dba64387847d68decb5791d3" candidate_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7728afd14434b77bd7881f41d26ba3f"
I20260619 01:03:11.645912 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11b29a20dba64387847d68decb5791d3" candidate_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
I20260619 01:03:11.645977 16288 raft_consensus.cc:3037] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:03:11.646122 16155 raft_consensus.cc:3037] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Advancing to term 1
W20260619 01:03:11.646128 16201 tablet.cc:2369] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:03:11.648778 16334 tablet.cc:2369] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:03:11.649256 16288 raft_consensus.cc:2454] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b86f8aad844b59abe8c8349ada0cc7 in term 1.
I20260619 01:03:11.649250 16155 raft_consensus.cc:2454] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b86f8aad844b59abe8c8349ada0cc7 in term 1.
I20260619 01:03:11.649619 15954 leader_election.cc:302] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8acd72c3e01e4959bcd0dcf0693bbce4, c6b86f8aad844b59abe8c8349ada0cc7; no voters:
I20260619 01:03:11.649905 16358 raft_consensus.cc:2781] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:03:11.650341 16358 raft_consensus.cc:686] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 LEADER]: Becoming Leader. State: Replica: c6b86f8aad844b59abe8c8349ada0cc7, State: Running, Role: LEADER
I20260619 01:03:11.650835 16358 consensus_queue.cc:237] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } }
I20260619 01:03:11.655330 15875 catalog_manager.cc:5495] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } health_report { overall_health: HEALTHY } } }
I20260619 01:03:11.683038 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.677869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.0.97.1:35395)
0619 01:03:11.678074 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.678080 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.678098 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:03:11.678257 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.678261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.678333 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.678484 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:03:11.678909 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.679610 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.681011 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.681394 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.681521 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.681983 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:03:11.682468 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:03:11.682625 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.682690 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.682798 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:03:11.690582 16155 raft_consensus.cc:1261] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Refusing update from remote peer c6b86f8aad844b59abe8c8349ada0cc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:03:11.690649 16288 raft_consensus.cc:1261] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Refusing update from remote peer c6b86f8aad844b59abe8c8349ada0cc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:03:11.691589 16362 consensus_queue.cc:1035] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:03:11.691898 16358 consensus_queue.cc:1035] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:03:11.702973 16366 mvcc.cc:205] Tried to move back new op lower bound from 7298379741955567616 to 7298379741805416448. Current Snapshot: MvccSnapshot[applied={T|T < 7298379741955567616}]
I20260619 01:03:11.716831 15877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:03:11.717111 15877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:03:11.720688 15877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Sending DeleteTablet for 3 replicas of tablet 11b29a20dba64387847d68decb5791d3
I20260619 01:03:11.722090 16268 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b29a20dba64387847d68decb5791d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46956
I20260619 01:03:11.722137 16001 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b29a20dba64387847d68decb5791d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57896
I20260619 01:03:11.722263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15938
I20260619 01:03:11.722693 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 11b29a20dba64387847d68decb5791d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59774
I20260619 01:03:11.727048 16377 tablet_replica.cc:330] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: stopping tablet replica
I20260619 01:03:11.727541 16377 raft_consensus.cc:2229] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:03:11.733429 16377 raft_consensus.cc:2258] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:11.735747 16377 ts_tablet_manager.cc:1905] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:11.735769 16375 tablet_replica.cc:330] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: stopping tablet replica
I20260619 01:03:11.736313 16375 raft_consensus.cc:2229] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:03:11.736810 16375 raft_consensus.cc:2258] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:11.737530 16376 tablet_replica.cc:330] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: stopping tablet replica
I20260619 01:03:11.737898 16376 raft_consensus.cc:2229] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:03:11.738513 16376 raft_consensus.cc:2258] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:11.740880 16375 ts_tablet_manager.cc:1905] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:11.742647 16377 ts_tablet_manager.cc:1918] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:11.742823 16377 log.cc:1196] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/11b29a20dba64387847d68decb5791d3
I20260619 01:03:11.743340 16377 ts_tablet_manager.cc:1939] T 11b29a20dba64387847d68decb5791d3 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting consensus metadata
I20260619 01:03:11.743952 16376 ts_tablet_manager.cc:1905] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:11.745077 15861 catalog_manager.cc:4841] TS 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651): tablet 11b29a20dba64387847d68decb5791d3 (table test-table [id=81aaf59b2d0646d58f5995cec0302f38]) successfully deleted
Jun 19 01:03:11 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:03:11.748937 16375 ts_tablet_manager.cc:1918] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:11.749109 16375 log.cc:1196] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/11b29a20dba64387847d68decb5791d3
I20260619 01:03:11.749648 16375 ts_tablet_manager.cc:1939] T 11b29a20dba64387847d68decb5791d3 P a7728afd14434b77bd7881f41d26ba3f: Deleting consensus metadata
I20260619 01:03:11.751184 16376 ts_tablet_manager.cc:1918] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:11.751384 16376 log.cc:1196] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/11b29a20dba64387847d68decb5791d3
I20260619 01:03:11.751753 15862 catalog_manager.cc:4841] TS a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485): tablet 11b29a20dba64387847d68decb5791d3 (table test-table [id=81aaf59b2d0646d58f5995cec0302f38]) successfully deleted
I20260619 01:03:11.751909 16376 ts_tablet_manager.cc:1939] T 11b29a20dba64387847d68decb5791d3 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting consensus metadata
I20260619 01:03:11.754245 15863 catalog_manager.cc:4841] TS c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395): tablet 11b29a20dba64387847d68decb5791d3 (table test-table [id=81aaf59b2d0646d58f5995cec0302f38]) successfully deleted
I20260619 01:03:11.759660 16350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.723630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57910 (local address 127.0.97.1:35395)
0619 01:03:11.723866 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.723892 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.736170 (+ 12278us) server_negotiation.cc:407] Connection header received
0619 01:03:11.736717 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.736726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.736836 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.736932 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:11.739660 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.740834 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.743054 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.743798 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.743899 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.751016 (+ 7117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:11.751031 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:11.751050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:11.751116 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:11.756532 (+ 5416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.757381 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.757388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.757390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.757536 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.758024 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.758028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.758030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.758404 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:11.758626 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.759073 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.759265 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:03:11.760442 16061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:11.866648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15938
I20260619 01:03:11.885258 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16072
Jun 19 01:03:11 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:03:11.900923 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:11.885957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.97.2:44651)
0619 01:03:11.886132 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:11.886139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:11.886205 (+ 66us) server_negotiation.cc:407] Connection header received
0619 01:03:11.886421 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:11.886426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:11.886496 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:11.886602 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:11.887887 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.888601 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.889885 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:11.890264 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:11.890428 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:11.895128 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:11.895139 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:11.895151 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:11.895193 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:11.898007 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.898700 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.898827 (+ 127us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.898831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.898948 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:11.899327 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:11.899334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:11.899337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:11.899722 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:11.900005 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:11.900360 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 01:03:11.900598 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:03:11.901620 16195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:11.993048 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16072
I20260619 01:03:12.009654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16205
Jun 19 01:03:12 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781830989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:03:12.024200 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:12.010459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.0.97.3:37485)
0619 01:03:12.010650 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:12.010658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:12.010696 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:03:12.010923 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:12.010929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:12.011015 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:12.011125 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:12.012397 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:12.013145 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:12.014493 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:12.015010 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:12.015143 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:12.019306 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:12.019321 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:12.019336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:12.019366 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:12.021923 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:12.022516 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:12.022523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:12.022525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:12.022649 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:12.022994 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:12.022998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:12.022999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:12.023314 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:12.023534 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:12.023791 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 01:03:12.023946 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:03:12.024923 16328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:12.113765 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16205
I20260619 01:03:12.130048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 15845
I20260619 01:03:12.131191 15907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:12.240602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 15845
I20260619 01:03:32.264123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--webserver_interface=127.0.97.62
--webserver_port=43913
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:32.582125 16383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:32.582459 16383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:32.582617 16383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:32.589910 16383 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:03:32.590036 16383 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:03:32.590128 16383 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:03:32.590229 16383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:03:32.590329 16383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:32.590463 16383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:32.590552 16383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:03:32.590632 16383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:03:32.600237 16383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:32.601487 16383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:32.602824 16383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:32.624830 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:32.624856 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:32.624905 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:32.625463 16383 server_base.cc:1034] running on GCE node
I20260619 01:03:32.626470 16383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:32.628087 16383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:32.629344 16383 hybrid_clock.cc:648] HybridClock initialized: now 1781831012629277 us; error 40 us; skew 500 ppm
I20260619 01:03:32.629721 16383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:32 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831012, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:32.633224 16383 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:03:32.634984 16383 webserver.cc:460] Webserver started at http://127.0.97.62:43913/ using document root <none> and password file <none>
I20260619 01:03:32.635756 16383 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:32.635892 16383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:32.641496 16383 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 01:03:32.644361 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:32.645635 16383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:03:32.645818 16383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "cbfdffe046fc47e9bc903b2f59a56d06"
format_stamp: "Formatted at 2026-06-19 01:03:09 on dist-test-slave-3rxd"
server_key: "d8c5b50921d82ef9374fccf2dc23b75c"
server_key_iv: "20326c005bf8fa18c2229e9534fa180e"
server_key_version: "encryptionkey@0"
I20260619 01:03:32.646857 16383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:32.662561 16383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:32.666651 16383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:32.666967 16383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:32.685144 16383 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35037
I20260619 01:03:32.685179 16450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35037 every 8 connection(s)
I20260619 01:03:32.686735 16383 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:03:32.693605 16451 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:03:32.695996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16383
I20260619 01:03:32.696836 16451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap starting.
I20260619 01:03:32.696858 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=36043
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:03:32.721366 16451 log.cc:824] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:32.737042 16451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:03:32.737679 16451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap complete.
I20260619 01:03:32.750808 16451 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:32.751960 16451 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Initialized, Role: FOLLOWER
I20260619 01:03:32.752650 16451 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:32.752928 16451 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:03:32.753074 16451 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:03:32.753283 16451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 1 FOLLOWER]: Advancing to term 2
I20260619 01:03:32.759035 16451 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:32.759660 16451 leader_election.cc:302] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfdffe046fc47e9bc903b2f59a56d06; no voters:
I20260619 01:03:32.760318 16451 leader_election.cc:288] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 01:03:32.760471 16455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 01:03:32.761850 16455 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 LEADER]: Becoming Leader. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Running, Role: LEADER
I20260619 01:03:32.762468 16451 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:03:32.762398 16455 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:32.764904 16457 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfdffe046fc47e9bc903b2f59a56d06. Latest consensus state: current_term: 2 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:32.765281 16457 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:32.764979 16456 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:32.765697 16456 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:32.767875 16462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:03:32.775777 16462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:03:32.777107 16462 catalog_manager.cc:1259] Loaded cluster ID: 076bfdc213964ce8b7c5b92aed3f28a1
I20260619 01:03:32.777242 16462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:03:32.781309 16462 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:03:32.782757 16462 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Loaded TSK: 0
I20260619 01:03:32.808624 16472 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 01:03:32.813556 16462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Generated new TSK 1
I20260619 01:03:32.813897 16462 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 01:03:33.055096 16453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:33.055349 16453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:33.055480 16453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:33.062463 16453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:33.062566 16453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:33.062805 16453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:03:33.072620 16453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36043
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:33.073915 16453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:33.075273 16453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:33.089284 16478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.089313 16479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.090070 16481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:33.090425 16453 server_base.cc:1034] running on GCE node
I20260619 01:03:33.091087 16453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:33.092051 16453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:33.093302 16453 hybrid_clock.cc:648] HybridClock initialized: now 1781831013093249 us; error 37 us; skew 500 ppm
I20260619 01:03:33.093622 16453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:33 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:33.096733 16453 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:03:33.098309 16453 webserver.cc:460] Webserver started at http://127.0.97.1:36043/ using document root <none> and password file <none>
I20260619 01:03:33.099036 16453 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:33.099179 16453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:33.104609 16453 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 01:03:33.107379 16488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.108832 16453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:03:33.108994 16453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "880a6e682bc00c1a8bb361652e2ed33b"
server_key_iv: "df18da0cf8786cc33b983486e1623a11"
server_key_version: "encryptionkey@0"
I20260619 01:03:33.109964 16453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:33.133698 16453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:33.137511 16453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:33.137774 16453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:33.138760 16453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:33.140025 16453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:33.140120 16453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.140267 16453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:33.140362 16453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.173748 16453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35395
I20260619 01:03:33.173849 16600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35395 every 8 connection(s)
I20260619 01:03:33.175300 16453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:03:33.185144 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16453
I20260619 01:03:33.185779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44651
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=34535
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:03:33 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:33.199375 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:33.178150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59795 (local address 127.0.97.62:35037)
0619 01:03:33.178667 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:33.178679 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:33.179484 (+ 805us) server_negotiation.cc:407] Connection header received
0619 01:03:33.181608 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:33.181634 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:33.181970 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:33.182272 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:33.183678 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.185207 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.186856 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.187973 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.188177 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:33.191536 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:33.191549 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:33.191570 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:33.191621 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:33.194912 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:33.195846 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:33.195853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:33.195858 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:33.196019 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:33.196512 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:33.196515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:33.196517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:33.196994 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:33.197134 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:33.198033 (+ 899us) server_negotiation.cc:299] Negotiation successful
0619 01:03:33.198341 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":209,"threads_started":1}
I20260619 01:03:33.199756 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:33.178420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:59795)
0619 01:03:33.179305 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:33.179351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:33.181111 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:33.182402 (+ 1291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:33.182410 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:33.182992 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:33.183513 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:33.183529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.185400 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.185404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:33.186657 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:33.186669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.188185 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.188190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:33.188380 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:33.189041 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:33.189063 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:33.191226 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:33.195044 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:33.195056 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:33.195074 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:33.195701 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:33.196140 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:33.196147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:33.196154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:33.196401 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:33.197133 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:33.197144 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:33.197566 (+ 422us) client_negotiation.cc:769] Sending connection context
0619 01:03:33.197850 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 01:03:33.198417 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":394,"threads_started":1}
I20260619 01:03:33.202989 16601 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:33.203320 16601 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:33.204028 16601 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260619 01:03:33.206169 16416 ts_manager.cc:194] Registered new tserver with Master: c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395)
I20260619 01:03:33.207726 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59795
W20260619 01:03:33.501636 16605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:33.501963 16605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:33.502127 16605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:33.509840 16605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:33.510007 16605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:33.510239 16605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:03:33.522151 16605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34535
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:33.523512 16605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:33.524770 16605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:33.537842 16610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.537818 16611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.537848 16613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:33.538507 16605 server_base.cc:1034] running on GCE node
I20260619 01:03:33.539220 16605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:33.540138 16605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:33.541364 16605 hybrid_clock.cc:648] HybridClock initialized: now 1781831013541313 us; error 45 us; skew 500 ppm
I20260619 01:03:33.541673 16605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:33 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:33.544667 16605 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:03:33.545950 16605 webserver.cc:460] Webserver started at http://127.0.97.2:34535/ using document root <none> and password file <none>
I20260619 01:03:33.546602 16605 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:33.546783 16605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:33.552009 16605 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 01:03:33.554517 16620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.555851 16605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:03:33.556021 16605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "2d586709e3ff0aaa00c8d197b476d266"
server_key_iv: "4e2e21106ac1a6850aa55e80b1939e1a"
server_key_version: "encryptionkey@0"
I20260619 01:03:33.556959 16605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:33.568447 16605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:33.572348 16605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:33.572598 16605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:33.573501 16605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:33.574805 16605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:33.574924 16605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.575053 16605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:33.575141 16605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.606722 16605 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44651
I20260619 01:03:33.606822 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44651 every 8 connection(s)
I20260619 01:03:33.608088 16605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:03:33.611083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16605
I20260619 01:03:33.611639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37485
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=43325
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 01:03:33 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:33.627841 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:33.610693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32933 (local address 127.0.97.62:35037)
0619 01:03:33.610945 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:33.610952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:33.612218 (+ 1266us) server_negotiation.cc:407] Connection header received
0619 01:03:33.614127 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:33.614135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:33.614226 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:33.614338 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:33.615823 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.616937 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.618409 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.618927 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.619075 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:33.622488 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:33.622493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:33.622496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:33.622527 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:33.624953 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:33.625653 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:33.625665 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:33.625668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:33.625787 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:33.626196 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:33.626204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:33.626206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:33.626455 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:33.626604 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:33.627272 (+ 668us) server_negotiation.cc:299] Negotiation successful
0619 01:03:33.627512 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 01:03:33.628753 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:33.611242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:32933)
0619 01:03:33.612049 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:33.612091 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:33.613834 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:33.614566 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:33.614574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:33.615107 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:33.615643 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:33.615660 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.617089 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.617093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:33.618268 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:33.618279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:33.619157 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:33.619166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:33.619432 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:33.620185 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:33.620207 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:33.622266 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:33.625086 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:33.625095 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:33.625110 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:33.625529 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:33.625902 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:33.625907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:33.625910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:33.626080 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:33.626581 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:33.626588 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:33.626947 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 01:03:33.627192 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 01:03:33.627677 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":359,"threads_started":1}
I20260619 01:03:33.630199 16733 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:33.630564 16733 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:33.631491 16733 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260619 01:03:33.633108 16416 ts_manager.cc:194] Registered new tserver with Master: 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651)
I20260619 01:03:33.633999 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:32933
W20260619 01:03:33.934125 16736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:33.934401 16736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:33.934592 16736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:33.941938 16736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:33.942047 16736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:33.942159 16736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:03:33.952112 16736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43325
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:33.953428 16736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:33.954797 16736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:33.968016 16742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.968175 16741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:33.968211 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:33.968993 16736 server_base.cc:1034] running on GCE node
I20260619 01:03:33.969570 16736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:33.970527 16736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:33.971730 16736 hybrid_clock.cc:648] HybridClock initialized: now 1781831013971705 us; error 57 us; skew 500 ppm
I20260619 01:03:33.972110 16736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:33 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:33.975243 16736 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:03:33.976631 16736 webserver.cc:460] Webserver started at http://127.0.97.3:43325/ using document root <none> and password file <none>
I20260619 01:03:33.977288 16736 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:33.977452 16736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:33.982544 16736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:03:33.985224 16751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:33.986488 16736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:03:33.986677 16736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "a7728afd14434b77bd7881f41d26ba3f"
format_stamp: "Formatted at 2026-06-19 01:03:11 on dist-test-slave-3rxd"
server_key: "e35f9acfc538b2c62f166a3fa17ee02e"
server_key_iv: "25cdb321cbf632158a0984fdd4b2cfe3"
server_key_version: "encryptionkey@0"
I20260619 01:03:33.987676 16736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:34.002363 16736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:34.006697 16736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:34.007038 16736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:34.007959 16736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:34.009227 16736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:34.009354 16736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:34.009486 16736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:34.009577 16736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:34.042272 16736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37485
I20260619 01:03:34.042351 16863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37485 every 8 connection(s)
I20260619 01:03:34.043677 16736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:03:34.045136 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16736
Jun 19 01:03:34 dist-test-slave-3rxd krb5kdc[15832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:03:34.061049 16867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.046199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36982 (local address 127.0.97.62:35037)
0619 01:03:34.046784 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.046792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.047119 (+ 327us) server_negotiation.cc:407] Connection header received
0619 01:03:34.047340 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.047343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.047406 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.047518 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.049056 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.049928 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.051616 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.052081 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.052214 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.054912 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.054917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.054920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.054950 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.057759 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.058568 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.058572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.058574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.058702 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.059134 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.059142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.059146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.059580 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.059850 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.060288 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.060561 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":436,"thread_start_us":239,"threads_started":1}
I20260619 01:03:34.064595 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.046199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40323 (local address 127.0.97.62:35037)
0619 01:03:34.046406 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.046414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.047950 (+ 1536us) server_negotiation.cc:407] Connection header received
0619 01:03:34.049625 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.049629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.049717 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.049807 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.051500 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.052286 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.053930 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.054322 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.054459 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.057982 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.057989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.057995 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.058050 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.061653 (+ 3603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.062447 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.062452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.062455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.062581 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.063063 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.063066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.063068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.063403 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.063536 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.064211 (+ 675us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.064364 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 01:03:34.065409 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.047024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:40323)
0619 01:03:34.047787 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.047829 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.049392 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.049987 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.049995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.050549 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:03:34.051334 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.051358 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.052452 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.052459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.053777 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.053789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.054544 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.054551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.054834 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.055467 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:03:34.055498 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:03:34.057832 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:03:34.061789 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:34.061799 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:34.061812 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:34.062318 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:34.062689 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:03:34.062693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:03:34.062696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:03:34.062961 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:03:34.063536 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:03:34.063543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:03:34.063883 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.064087 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.064415 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":325,"threads_started":1}
I20260619 01:03:34.066556 16864 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35037
I20260619 01:03:34.066879 16864 heartbeater.cc:464] Registering TS with master...
I20260619 01:03:34.067526 16864 heartbeater.cc:511] Master 127.0.97.62:35037 requested a full tablet report, sending...
I20260619 01:03:34.068898 16416 ts_manager.cc:194] Registered new tserver with Master: a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485)
I20260619 01:03:34.069723 16416 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40323
I20260619 01:03:34.075651 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:03:34.081332 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.076507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.97.62:35037)
0619 01:03:34.076639 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.076643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.077073 (+ 430us) server_negotiation.cc:407] Connection header received
0619 01:03:34.077219 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.077223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.077332 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.077440 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:03:34.077812 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.078502 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.079837 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.080221 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.080317 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.080768 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:03:34.080929 (+ 161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 01:03:34.081159 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36986: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 01:03:34.081455 16603 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36986: token expired
W20260619 01:03:34.082001 16337 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:35037)
I20260619 01:03:34.097503 16603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.082437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.0.97.62:35037)
0619 01:03:34.082588 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.082594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.082664 (+ 70us) server_negotiation.cc:407] Connection header received
0619 01:03:34.082888 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.082897 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.082958 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.083052 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.084338 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.085005 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.086238 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.086615 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.086778 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.092660 (+ 5882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.092665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.092668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.092696 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.094996 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.095624 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.095631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.095639 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.095770 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.096160 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.096166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.096169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.096456 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.096696 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.097033 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.097235 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:03:34.100620 16416 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:03:34.107751 16416 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:03:34.142444 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.119031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.97.2:44651)
0619 01:03:34.129906 (+ 10875us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.129918 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.133654 (+ 3736us) server_negotiation.cc:407] Connection header received
0619 01:03:34.133955 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.133965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.134399 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.134617 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:34.135541 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.136993 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.139891 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141383 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141511 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.141702 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.141983 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.142133 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10717,"thread_start_us":10630,"threads_started":1}
I20260619 01:03:34.142550 16874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.118686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44651 (local address 127.0.0.1:33048)
0619 01:03:34.133498 (+ 14812us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.133536 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.133731 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.134828 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.134834 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.134861 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:34.135276 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.135292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.137127 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.137133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.139712 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.139728 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141560 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.141722 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.141797 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.141925 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.142118 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14601,"mutex_wait_us":4,"thread_start_us":14409,"threads_started":1}
I20260619 01:03:34.142902 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.119051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.0.97.1:35395)
0619 01:03:34.131925 (+ 12874us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.131938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.134462 (+ 2524us) server_negotiation.cc:407] Connection header received
0619 01:03:34.134804 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.134811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.135040 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.135202 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:34.136126 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.137484 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.140171 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141550 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141656 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.141819 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.142386 (+ 567us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.142551 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12678,"thread_start_us":12909,"threads_started":1}
I20260619 01:03:34.143253 16878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.118823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35395 (local address 127.0.0.1:43592)
0619 01:03:34.134349 (+ 15526us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.134366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.134604 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.135516 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.135522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.135544 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:34.135914 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.135925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.137645 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.137650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.139935 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.139944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141970 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.142163 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.142197 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.142340 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.142462 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15391,"mutex_wait_us":39,"thread_start_us":15636,"threads_started":1}
I20260619 01:03:34.144347 16875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.120611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37485 (local address 127.0.0.1:51370)
0619 01:03:34.134762 (+ 14151us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.134778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.134975 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.135808 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.135811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.135832 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:34.136218 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.136231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.137927 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.137933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.139888 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.139901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141492 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.143758 (+ 2258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.143794 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.143893 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.144008 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13988,"mutex_wait_us":1172,"thread_start_us":13450,"threads_started":1}
I20260619 01:03:34.144433 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.120969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.0.97.3:37485)
0619 01:03:34.132915 (+ 11946us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.132921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.134872 (+ 1951us) server_negotiation.cc:407] Connection header received
0619 01:03:34.135169 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.135177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.135395 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.135570 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:34.136398 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.137738 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.140040 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.141009 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.141155 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.141400 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.143956 (+ 2556us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.144107 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11816,"thread_start_us":11374,"threads_started":1}
I20260619 01:03:34.146302 16668 tablet_service.cc:1467] Processing CreateTablet for tablet 82b230f1924d4b68851a95e597a3b923 (DEFAULT_TABLE table=test-table [id=d2f3716ff265436c9e2b058272879f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:34.146669 16536 tablet_service.cc:1467] Processing CreateTablet for tablet 82b230f1924d4b68851a95e597a3b923 (DEFAULT_TABLE table=test-table [id=d2f3716ff265436c9e2b058272879f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:34.147742 16668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b230f1924d4b68851a95e597a3b923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:34.148048 16536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b230f1924d4b68851a95e597a3b923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:34.147980 16799 tablet_service.cc:1467] Processing CreateTablet for tablet 82b230f1924d4b68851a95e597a3b923 (DEFAULT_TABLE table=test-table [id=d2f3716ff265436c9e2b058272879f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:03:34.149271 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82b230f1924d4b68851a95e597a3b923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:03:34.159770 16880 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Bootstrap starting.
I20260619 01:03:34.162889 16881 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Bootstrap starting.
I20260619 01:03:34.163471 16880 tablet_bootstrap.cc:654] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:34.164527 16882 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Bootstrap starting.
I20260619 01:03:34.166102 16880 log.cc:824] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:34.166301 16881 tablet_bootstrap.cc:654] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:34.167443 16882 tablet_bootstrap.cc:654] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Neither blocks nor log segments found. Creating new log.
I20260619 01:03:34.167421 16881 log.cc:824] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:34.168471 16882 log.cc:824] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:34.169551 16880 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: No bootstrap required, opened a new log
I20260619 01:03:34.169878 16880 ts_tablet_manager.cc:1397] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 01:03:34.171100 16882 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: No bootstrap required, opened a new log
I20260619 01:03:34.171356 16882 ts_tablet_manager.cc:1397] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 01:03:34.173481 16881 tablet_bootstrap.cc:492] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: No bootstrap required, opened a new log
I20260619 01:03:34.173774 16881 ts_tablet_manager.cc:1397] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20260619 01:03:34.181660 16882 raft_consensus.cc:348] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.181671 16880 raft_consensus.cc:348] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.182070 16882 raft_consensus.cc:374] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:34.182075 16880 raft_consensus.cc:374] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:34.182183 16880 raft_consensus.cc:729] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b86f8aad844b59abe8c8349ada0cc7, State: Initialized, Role: FOLLOWER
I20260619 01:03:34.182178 16882 raft_consensus.cc:729] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7728afd14434b77bd7881f41d26ba3f, State: Initialized, Role: FOLLOWER
I20260619 01:03:34.182884 16882 consensus_queue.cc:260] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.182901 16880 consensus_queue.cc:260] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.185113 16881 raft_consensus.cc:348] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.185693 16881 raft_consensus.cc:374] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:03:34.185853 16881 raft_consensus.cc:729] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acd72c3e01e4959bcd0dcf0693bbce4, State: Initialized, Role: FOLLOWER
I20260619 01:03:34.186714 16881 consensus_queue.cc:260] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.190331 16864 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260619 01:03:34.190999 16882 ts_tablet_manager.cc:1428] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
I20260619 01:03:34.192135 16881 ts_tablet_manager.cc:1428] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Time spent starting tablet: real 0.018s user 0.009s sys 0.006s
I20260619 01:03:34.192605 16733 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
I20260619 01:03:34.192790 16880 ts_tablet_manager.cc:1428] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Time spent starting tablet: real 0.023s user 0.013s sys 0.008s
I20260619 01:03:34.193276 16601 heartbeater.cc:503] Master 127.0.97.62:35037 was elected leader, sending a full tablet report...
W20260619 01:03:34.295207 16865 tablet.cc:2369] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:03:34.328466 16886 raft_consensus.cc:482] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:03:34.328748 16886 raft_consensus.cc:504] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.330340 16886 leader_election.cc:288] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651), c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395)
I20260619 01:03:34.337273 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.330744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44815 (local address 127.0.97.2:44651)
0619 01:03:34.330955 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.330963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.330988 (+ 25us) server_negotiation.cc:407] Connection header received
0619 01:03:34.331241 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.331246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.331327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.331428 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:34.331873 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.332783 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.335306 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.336498 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.336631 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.336725 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.336939 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.337050 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:03:34.337517 16868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.330574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44651 (local address 127.0.97.3:44815)
0619 01:03:34.330890 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.330909 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.331028 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.331460 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.331466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.331492 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:34.331722 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.331734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.332930 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.332936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.335169 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.335182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.336644 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.336648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.336768 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.336790 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.336905 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.337040 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":111}
I20260619 01:03:34.337586 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.330779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35395 (local address 127.0.97.3:41529)
0619 01:03:34.331540 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:34.331570 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 01:03:34.331679 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:03:34.332010 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:03:34.332012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:03:34.332030 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:03:34.332306 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.332313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.333375 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.333380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.335397 (+ 2017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:03:34.335406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.336937 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.336943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:03:34.337084 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.337111 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 01:03:34.337201 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 01:03:34.337299 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":423,"threads_started":1}
I20260619 01:03:34.337607 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.330840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41529 (local address 127.0.97.1:35395)
0619 01:03:34.331017 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.331022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.331657 (+ 635us) server_negotiation.cc:407] Connection header received
0619 01:03:34.331800 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.331824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.331897 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.331993 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:03:34.332446 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.333249 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.335555 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.336747 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.336854 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.336960 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.337236 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.337363 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:03:34.338606 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b230f1924d4b68851a95e597a3b923" candidate_uuid: "a7728afd14434b77bd7881f41d26ba3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" is_pre_election: true
I20260619 01:03:34.338609 16556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b230f1924d4b68851a95e597a3b923" candidate_uuid: "a7728afd14434b77bd7881f41d26ba3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" is_pre_election: true
I20260619 01:03:34.339154 16556 raft_consensus.cc:2454] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7728afd14434b77bd7881f41d26ba3f in term 0.
I20260619 01:03:34.339154 16688 raft_consensus.cc:2454] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7728afd14434b77bd7881f41d26ba3f in term 0.
I20260619 01:03:34.339866 16754 leader_election.cc:302] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8acd72c3e01e4959bcd0dcf0693bbce4, a7728afd14434b77bd7881f41d26ba3f, c6b86f8aad844b59abe8c8349ada0cc7; no voters:
I20260619 01:03:34.340278 16886 raft_consensus.cc:2781] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:03:34.340425 16886 raft_consensus.cc:482] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:03:34.340533 16886 raft_consensus.cc:3037] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:03:34.342499 16886 raft_consensus.cc:504] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.343147 16886 leader_election.cc:288] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [CANDIDATE]: Term 1 election: Requested vote from peers 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651), c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395)
I20260619 01:03:34.343475 16688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b230f1924d4b68851a95e597a3b923" candidate_uuid: "a7728afd14434b77bd7881f41d26ba3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
I20260619 01:03:34.343573 16556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82b230f1924d4b68851a95e597a3b923" candidate_uuid: "a7728afd14434b77bd7881f41d26ba3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
I20260619 01:03:34.343669 16688 raft_consensus.cc:3037] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:03:34.343771 16556 raft_consensus.cc:3037] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:03:34.346756 16556 raft_consensus.cc:2454] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7728afd14434b77bd7881f41d26ba3f in term 1.
I20260619 01:03:34.346748 16688 raft_consensus.cc:2454] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7728afd14434b77bd7881f41d26ba3f in term 1.
I20260619 01:03:34.347124 16752 leader_election.cc:302] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8acd72c3e01e4959bcd0dcf0693bbce4, a7728afd14434b77bd7881f41d26ba3f, c6b86f8aad844b59abe8c8349ada0cc7; no voters:
I20260619 01:03:34.347522 16886 raft_consensus.cc:2781] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:03:34.347956 16886 raft_consensus.cc:686] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 1 LEADER]: Becoming Leader. State: Replica: a7728afd14434b77bd7881f41d26ba3f, State: Running, Role: LEADER
I20260619 01:03:34.348428 16886 consensus_queue.cc:237] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } }
I20260619 01:03:34.352895 16415 catalog_manager.cc:5495] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f reported cstate change: term changed from 0 to 1, leader changed from <none> to a7728afd14434b77bd7881f41d26ba3f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a7728afd14434b77bd7881f41d26ba3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7728afd14434b77bd7881f41d26ba3f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37485 } health_report { overall_health: HEALTHY } } }
W20260619 01:03:34.360301 16734 tablet.cc:2369] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:03:34.380218 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.375418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51374 (local address 127.0.97.3:37485)
0619 01:03:34.375604 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.375609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.375649 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:03:34.375824 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.375828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.375899 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.376029 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:03:34.376346 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.377079 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.378307 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.378818 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.378913 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.379326 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:03:34.379784 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:03:34.379888 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.379973 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.380035 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:03:34.387821 16688 raft_consensus.cc:1261] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Refusing update from remote peer a7728afd14434b77bd7881f41d26ba3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:03:34.387820 16556 raft_consensus.cc:1261] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Refusing update from remote peer a7728afd14434b77bd7881f41d26ba3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:03:34.388866 16886 consensus_queue.cc:1035] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8acd72c3e01e4959bcd0dcf0693bbce4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:03:34.389150 16890 consensus_queue.cc:1035] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b86f8aad844b59abe8c8349ada0cc7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:03:34.416813 16416 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:03:34.417089 16416 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:03:34.420385 16416 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Sending DeleteTablet for 3 replicas of tablet 82b230f1924d4b68851a95e597a3b923
I20260619 01:03:34.421545 16668 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b230f1924d4b68851a95e597a3b923 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33048
I20260619 01:03:34.421562 16536 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b230f1924d4b68851a95e597a3b923 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43592
I20260619 01:03:34.422408 16903 tablet_replica.cc:330] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: stopping tablet replica
I20260619 01:03:34.422552 16799 tablet_service.cc:1514] Processing DeleteTablet for tablet 82b230f1924d4b68851a95e597a3b923 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:03:34 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51370
I20260619 01:03:34.422850 16903 raft_consensus.cc:2229] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:03:34.422932 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16453
I20260619 01:03:34.423296 16903 raft_consensus.cc:2258] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:34.423990 16904 tablet_replica.cc:330] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: stopping tablet replica
I20260619 01:03:34.424007 16905 tablet_replica.cc:330] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: stopping tablet replica
I20260619 01:03:34.424361 16904 raft_consensus.cc:2229] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:03:34.424407 16905 raft_consensus.cc:2229] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:03:34.424782 16904 raft_consensus.cc:2258] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:34.425230 16905 raft_consensus.cc:2258] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:03:34.427273 16905 ts_tablet_manager.cc:1905] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:34.428509 16903 ts_tablet_manager.cc:1905] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:34.433815 16904 ts_tablet_manager.cc:1905] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:03:34.434615 16905 ts_tablet_manager.cc:1918] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:34.434842 16905 log.cc:1196] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/82b230f1924d4b68851a95e597a3b923
I20260619 01:03:34.435387 16905 ts_tablet_manager.cc:1939] T 82b230f1924d4b68851a95e597a3b923 P a7728afd14434b77bd7881f41d26ba3f: Deleting consensus metadata
I20260619 01:03:34.437232 16401 catalog_manager.cc:4841] TS a7728afd14434b77bd7881f41d26ba3f (127.0.97.3:37485): tablet 82b230f1924d4b68851a95e597a3b923 (table test-table [id=d2f3716ff265436c9e2b058272879f75]) successfully deleted
I20260619 01:03:34.439110 16903 ts_tablet_manager.cc:1918] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:34.439260 16903 log.cc:1196] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/82b230f1924d4b68851a95e597a3b923
I20260619 01:03:34.439759 16903 ts_tablet_manager.cc:1939] T 82b230f1924d4b68851a95e597a3b923 P c6b86f8aad844b59abe8c8349ada0cc7: Deleting consensus metadata
I20260619 01:03:34.440418 16904 ts_tablet_manager.cc:1918] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:03:34.440562 16904 log.cc:1196] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/82b230f1924d4b68851a95e597a3b923
I20260619 01:03:34.441061 16904 ts_tablet_manager.cc:1939] T 82b230f1924d4b68851a95e597a3b923 P 8acd72c3e01e4959bcd0dcf0693bbce4: Deleting consensus metadata
I20260619 01:03:34.441478 16402 catalog_manager.cc:4841] TS c6b86f8aad844b59abe8c8349ada0cc7 (127.0.97.1:35395): tablet 82b230f1924d4b68851a95e597a3b923 (table test-table [id=d2f3716ff265436c9e2b058272879f75]) successfully deleted
I20260619 01:03:34.442570 16400 catalog_manager.cc:4841] TS 8acd72c3e01e4959bcd0dcf0693bbce4 (127.0.97.2:44651): tablet 82b230f1924d4b68851a95e597a3b923 (table test-table [id=d2f3716ff265436c9e2b058272879f75]) successfully deleted
I20260619 01:03:34.443090 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.424415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.0.97.1:35395)
0619 01:03:34.424574 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.424579 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.424595 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:03:34.424650 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.424654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.424733 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.424843 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.426908 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.428053 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.430021 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.430675 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.430850 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.434515 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.434528 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.434544 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.434581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.438775 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.439976 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.439987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.439994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.440145 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.440744 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.440761 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.440764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.441311 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.441566 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.442649 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.442866 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:03:34.443734 16596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:34.531714 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16453
I20260619 01:03:34.554841 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16605
I20260619 01:03:34.570353 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.556764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.0.97.2:44651)
0619 01:03:34.557003 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.557008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.557024 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:03:34.557079 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.557082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.557141 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.557272 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.558603 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.559487 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.561030 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.561432 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.561558 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.564634 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.564646 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.564662 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.564692 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.567486 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.568225 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.568233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.568235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.568396 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.568790 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.568797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.568800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.569222 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.569531 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.569794 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.570005 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:03:34.571091 16728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:34.658587 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16605
I20260619 01:03:34.682883 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16736
I20260619 01:03:34.700688 16879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:34.683594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.97.3:37485)
0619 01:03:34.683779 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 01:03:34.683787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:03:34.683807 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:03:34.684021 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:03:34.684031 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:03:34.684099 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:03:34.684201 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:03:34.685878 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.687342 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.689233 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:03:34.689915 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:03:34.690049 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:03:34.693595 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:03:34.693609 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:03:34.693627 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:03:34.693671 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:03:34.697476 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.698340 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.698349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.698353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.698540 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:03:34.698971 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:03:34.698975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:03:34.698978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:03:34.699410 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:03:34.699706 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 01:03:34.700038 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 01:03:34.700283 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:03:34.701400 16859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:34.823240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16736
I20260619 01:03:34.843830 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16383
I20260619 01:03:34.844956 16446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:03:34.956230 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16383
I20260619 01:03:54.978575 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--webserver_interface=127.0.97.62
--webserver_port=43913
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:55.297961 16910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:55.298271 16910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:55.298424 16910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:55.305231 16910 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 01:03:55.305353 16910 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 01:03:55.305419 16910 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 01:03:55.305513 16910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:03:55.305613 16910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:55.305713 16910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:55.305796 16910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:03:55.305883 16910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:03:55.315384 16910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=43913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:55.316588 16910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:55.317894 16910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:55.329735 16915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:55.329771 16916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:55.329811 16918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:55.330325 16910 server_base.cc:1034] running on GCE node
I20260619 01:03:55.331123 16910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:55.332433 16910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:55.333668 16910 hybrid_clock.cc:648] HybridClock initialized: now 1781831035333606 us; error 39 us; skew 500 ppm
I20260619 01:03:55.334069 16910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:55 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:55.341773 16910 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:03:55.343245 16910 webserver.cc:460] Webserver started at http://127.0.97.62:43913/ using document root <none> and password file <none>
I20260619 01:03:55.343927 16910 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:55.344054 16910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:55.349246 16910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 01:03:55.351979 16925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:55.353193 16910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:03:55.353374 16910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "cbfdffe046fc47e9bc903b2f59a56d06"
format_stamp: "Formatted at 2026-06-19 01:03:09 on dist-test-slave-3rxd"
server_key: "d8c5b50921d82ef9374fccf2dc23b75c"
server_key_iv: "20326c005bf8fa18c2229e9534fa180e"
server_key_version: "encryptionkey@0"
I20260619 01:03:55.354292 16910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:55.375355 16910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:55.379022 16910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:55.379309 16910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:55.397490 16910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35037
I20260619 01:03:55.397567 16976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35037 every 8 connection(s)
I20260619 01:03:55.398897 16910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:03:55.400148 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16910
I20260619 01:03:55.400923 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35395
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=36043
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:03:55.405759 16977 sys_catalog.cc:263] Verifying existing consensus state
I20260619 01:03:55.408881 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap starting.
I20260619 01:03:55.428959 16977 log.cc:824] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Log is configured to *not* fsync() on all Append() calls
I20260619 01:03:55.444523 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 01:03:55.445015 16977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Bootstrap complete.
I20260619 01:03:55.453864 16977 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:55.454758 16977 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Initialized, Role: FOLLOWER
I20260619 01:03:55.455389 16977 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:55.455631 16977 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:03:55.455752 16977 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:03:55.455946 16977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 2 FOLLOWER]: Advancing to term 3
I20260619 01:03:55.459898 16977 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:55.460361 16977 leader_election.cc:302] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbfdffe046fc47e9bc903b2f59a56d06; no voters:
I20260619 01:03:55.461021 16977 leader_election.cc:288] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 01:03:55.461146 16981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 01:03:55.463315 16977 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:03:55.468238 16981 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [term 3 LEADER]: Becoming Leader. State: Replica: cbfdffe046fc47e9bc903b2f59a56d06, State: Running, Role: LEADER
I20260619 01:03:55.468928 16981 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } }
I20260619 01:03:55.471479 16982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:55.471875 16982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:55.472424 16983 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbfdffe046fc47e9bc903b2f59a56d06. Latest consensus state: current_term: 3 leader_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbfdffe046fc47e9bc903b2f59a56d06" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35037 } } }
I20260619 01:03:55.472774 16983 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:03:55.474612 16991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:03:55.490836 16991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:03:55.492384 16991 catalog_manager.cc:1259] Loaded cluster ID: 076bfdc213964ce8b7c5b92aed3f28a1
I20260619 01:03:55.492561 16991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:03:55.496500 16991 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:03:55.498142 16991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Loaded TSK: 0
I20260619 01:03:55.498373 16991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cbfdffe046fc47e9bc903b2f59a56d06: Loaded TSK: 1
F20260619 01:03:55.499550 16991 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f28a15179ad google::LogMessage::Fail() at ??:0
@ 0x7f28a151a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f28a15174de google::LogMessage::Flush() at ??:0
@ 0x7f28a1518a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f28a527b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f28a526a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f28b6f461c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f28b6f45783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f28b6fd12a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f28b6fd10c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f28b701b5e9 std::function<>::operator()() at ??:0
@ 0x7f28b6f49cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f28b6f47c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f28b6fcf610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f28b6fcf439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f28a4163345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f28a416691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f28a4166799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f28a4141eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f28a741b6db start_thread at ??:0
@ 0x7f28a03b161f clone at ??:0
W20260619 01:03:55.781067 16979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:55.781348 16979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:55.781505 16979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:55.788709 16979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:55.788820 16979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:55.789007 16979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:03:55.799331 16979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36043
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:55.800654 16979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:55.802039 16979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:55.815347 17024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:55.815292 17027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:55.815385 17025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:55.816205 16979 server_base.cc:1034] running on GCE node
I20260619 01:03:55.816857 16979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:55.817726 16979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:55.818941 16979 hybrid_clock.cc:648] HybridClock initialized: now 1781831035818895 us; error 42 us; skew 500 ppm
I20260619 01:03:55.819245 16979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:55 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:55.822155 16979 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:03:55.823647 16979 webserver.cc:460] Webserver started at http://127.0.97.1:36043/ using document root <none> and password file <none>
I20260619 01:03:55.824312 16979 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:55.824465 16979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:55.829535 16979 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 01:03:55.832125 17034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:55.833518 16979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:03:55.833693 16979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c6b86f8aad844b59abe8c8349ada0cc7"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "880a6e682bc00c1a8bb361652e2ed33b"
server_key_iv: "df18da0cf8786cc33b983486e1623a11"
server_key_version: "encryptionkey@0"
I20260619 01:03:55.834749 16979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:55.872774 16979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:55.877137 16979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:55.877413 16979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:55.878389 16979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:55.879706 16979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:55.879833 16979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:55.879973 16979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:55.880060 16979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:55.915424 16979 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35395
I20260619 01:03:55.915512 17146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35395 every 8 connection(s)
I20260619 01:03:55.916914 16979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:03:55.921878 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:55.920059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:47033)
0619 01:03:55.920794 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:55.921012 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":277,"threads_started":1}
W20260619 01:03:55.922540 17147 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35037 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35037: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
I20260619 01:03:55.923372 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:55.923082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:58257)
0619 01:03:55.923207 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:55.923272 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 01:03:55.924264 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:55.923945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:52609)
0619 01:03:55.924082 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:55.924163 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 01:03:55.924368 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 16979
W20260619 01:03:55.924491 17147 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:03:55.925041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44651
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=34535
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:03:56.242331 17150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:56.242591 17150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:56.242678 17150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:56.249289 17150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:56.249377 17150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:56.249476 17150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:03:56.259143 17150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44651
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34535
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:56.260337 17150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:56.261526 17150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:56.273854 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:56.273900 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:56.273927 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:56.274578 17150 server_base.cc:1034] running on GCE node
I20260619 01:03:56.275171 17150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:56.276259 17150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:56.277484 17150 hybrid_clock.cc:648] HybridClock initialized: now 1781831036277441 us; error 53 us; skew 500 ppm
I20260619 01:03:56.277776 17150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:56 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:56.280748 17150 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:03:56.282042 17150 webserver.cc:460] Webserver started at http://127.0.97.2:34535/ using document root <none> and password file <none>
I20260619 01:03:56.282790 17150 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:56.282924 17150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:56.287786 17150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 01:03:56.290223 17165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.291491 17150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 01:03:56.291659 17150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "8acd72c3e01e4959bcd0dcf0693bbce4"
format_stamp: "Formatted at 2026-06-19 01:03:10 on dist-test-slave-3rxd"
server_key: "2d586709e3ff0aaa00c8d197b476d266"
server_key_iv: "4e2e21106ac1a6850aa55e80b1939e1a"
server_key_version: "encryptionkey@0"
I20260619 01:03:56.292585 17150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:56.323822 17150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:56.327524 17150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:56.327791 17150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:56.328725 17150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:56.329933 17150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:56.330049 17150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.330175 17150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:56.330263 17150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.363293 17150 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44651
I20260619 01:03:56.363406 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44651 every 8 connection(s)
I20260619 01:03:56.364786 17150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:03:56.367259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17150
I20260619 01:03:56.367851 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37485
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:35037
--webserver_port=43325
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:03:56.370112 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.367767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:48995)
0619 01:03:56.368687 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.368951 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":329,"threads_started":1}
W20260619 01:03:56.370793 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35037 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35037: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
I20260619 01:03:56.371690 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.371264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:33603)
0619 01:03:56.371449 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.371539 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:03:56.372632 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.372192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:58997)
0619 01:03:56.372361 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.372467 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:03:56.372884 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 01:03:56.682818 17281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:03:56.683053 17281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:03:56.683233 17281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:03:56.690462 17281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:03:56.690596 17281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:03:56.690888 17281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:03:56.700961 17281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43325
--tserver_master_addrs=127.0.97.62:35037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:03:56.702164 17281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:03:56.703552 17281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:03:56.717235 17287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:56.717404 17286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:03:56.717420 17289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:03:56.723341 17281 server_base.cc:1034] running on GCE node
I20260619 01:03:56.723975 17281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:03:56.724864 17281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:03:56.726104 17281 hybrid_clock.cc:648] HybridClock initialized: now 1781831036726058 us; error 32 us; skew 500 ppm
I20260619 01:03:56.726449 17281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:03:56 dist-test-slave-3rxd krb5kdc[15832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:03:56.729523 17281 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:03:56.731058 17281 webserver.cc:460] Webserver started at http://127.0.97.3:43325/ using document root <none> and password file <none>
I20260619 01:03:56.731769 17281 fs_manager.cc:359] Metadata directory not provided
I20260619 01:03:56.731909 17281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:03:56.737062 17281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 01:03:56.739773 17296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.741030 17281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:03:56.741199 17281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "a7728afd14434b77bd7881f41d26ba3f"
format_stamp: "Formatted at 2026-06-19 01:03:11 on dist-test-slave-3rxd"
server_key: "e35f9acfc538b2c62f166a3fa17ee02e"
server_key_iv: "25cdb321cbf632158a0984fdd4b2cfe3"
server_key_version: "encryptionkey@0"
I20260619 01:03:56.742121 17281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:03:56.760373 17281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:03:56.764122 17281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:03:56.764426 17281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:03:56.765339 17281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:03:56.766598 17281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:03:56.766712 17281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.766861 17281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:03:56.766964 17281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:03:56.799582 17281 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37485
I20260619 01:03:56.799667 17408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37485 every 8 connection(s)
I20260619 01:03:56.801013 17281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:03:56.805783 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.804005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:53433)
0619 01:03:56.804697 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.804897 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":287,"threads_started":1}
W20260619 01:03:56.806423 17409 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:35037 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:35037: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
I20260619 01:03:56.807152 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.806875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:36883)
0619 01:03:56.806995 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.807058 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 01:03:56.807945 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.807620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:60149)
0619 01:03:56.807750 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.807850 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 01:03:56.808171 17409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 01:03:56.810849 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17281
I20260619 01:03:56.925652 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:56.924932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:59503)
0619 01:03:56.925360 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:56.925439 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260619 01:03:57.374074 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:57.373356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:44965)
0619 01:03:57.373763 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:57.373871 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":199,"threads_started":1}
I20260619 01:03:57.809475 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:57.808761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:59213)
0619 01:03:57.809177 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:57.809255 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":204,"threads_started":1}
I20260619 01:03:57.927145 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:57.926441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:42257)
0619 01:03:57.926912 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:57.926992 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":210,"threads_started":1}
I20260619 01:03:58.375526 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:58.374899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:43343)
0619 01:03:58.375295 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:58.375365 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":206,"threads_started":1}
I20260619 01:03:58.811188 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:58.810356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:49857)
0619 01:03:58.810929 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:58.811023 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":280,"threads_started":1}
I20260619 01:03:58.928534 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:58.927899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:47209)
0619 01:03:58.928302 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:58.928377 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":195,"threads_started":1}
I20260619 01:03:59.377046 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:59.376300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:48483)
0619 01:03:59.376788 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:59.376882 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
I20260619 01:03:59.812590 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:59.811987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:54921)
0619 01:03:59.812381 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:59.812449 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":199,"threads_started":1}
I20260619 01:03:59.929983 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:03:59.929362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:48211)
0619 01:03:59.929752 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 01:03:59.929827 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":196,"threads_started":1}
I20260619 01:04:00.378496 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:00.377841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:56495)
0619 01:04:00.378251 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:00.378323 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260619 01:04:00.813998 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:00.813372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:50557)
0619 01:04:00.813748 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:00.813819 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":183,"threads_started":1}
I20260619 01:04:00.931767 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:00.930922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:58413)
0619 01:04:00.931435 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:00.931531 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":261,"threads_started":1}
I20260619 01:04:01.380246 17426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:01.379461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:35967)
0619 01:04:01.379966 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:01.380072 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260619 01:04:01.815613 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:01.814899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:51623)
0619 01:04:01.815371 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:01.815449 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":248,"threads_started":1}
I20260619 01:04:01.933511 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:01.932712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:38331)
0619 01:04:01.933251 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:01.933349 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":318,"threads_started":1}
I20260619 01:04:02.381816 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:02.381119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:50675)
0619 01:04:02.381566 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:02.381655 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":237,"threads_started":1}
I20260619 01:04:02.817113 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:02.816431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:56481)
0619 01:04:02.816861 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:02.816943 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260619 01:04:02.935388 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:02.934418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:40601)
0619 01:04:02.934976 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:02.935072 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":265,"threads_started":1}
I20260619 01:04:03.383622 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:03.382779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:58117)
0619 01:04:03.383336 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:03.383425 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":254,"threads_started":1}
I20260619 01:04:03.819060 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:03.817908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:44377)
0619 01:04:03.818634 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:03.818784 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":375,"threads_started":1}
I20260619 01:04:03.937016 17434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:03.936226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:45421)
0619 01:04:03.936699 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:03.936815 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":218,"threads_started":1}
I20260619 01:04:04.385106 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:04.384458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:54359)
0619 01:04:04.384883 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:04.384956 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":244,"threads_started":1}
I20260619 01:04:04.820713 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:04.819975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:50561)
0619 01:04:04.820482 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:04.820557 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":290,"threads_started":1}
I20260619 01:04:04.938503 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:04.937789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:44693)
0619 01:04:04.938265 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:04.938336 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":242,"threads_started":1}
I20260619 01:04:05.386596 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:05.385918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:59143)
0619 01:04:05.386364 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:05.386436 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
I20260619 01:04:05.822407 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:05.821686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:46087)
0619 01:04:05.822158 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:05.822237 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260619 01:04:05.939978 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:05.939355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:55307)
0619 01:04:05.939756 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:05.939829 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":224,"threads_started":1}
I20260619 01:04:06.388192 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:06.387466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:47877)
0619 01:04:06.387937 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:06.388025 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":268,"threads_started":1}
I20260619 01:04:06.824018 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:06.823285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:37233)
0619 01:04:06.823754 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:06.823860 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":276,"threads_started":1}
I20260619 01:04:06.941496 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:06.940761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:54011)
0619 01:04:06.941225 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:06.941310 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":258,"threads_started":1}
I20260619 01:04:07.389703 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:07.389067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:33633)
0619 01:04:07.389452 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:07.389535 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":195,"threads_started":1}
I20260619 01:04:07.825724 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:07.824919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:60355)
0619 01:04:07.825457 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:07.825558 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":265,"threads_started":1}
I20260619 01:04:07.943035 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:07.942305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:60379)
0619 01:04:07.942792 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:07.942878 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260619 01:04:08.391426 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:08.390633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:49241)
0619 01:04:08.391188 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:08.391263 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":329,"threads_started":1}
I20260619 01:04:08.827236 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:08.826569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:47399)
0619 01:04:08.827014 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:08.827087 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
I20260619 01:04:08.944640 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:08.943823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:43781)
0619 01:04:08.944335 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:08.944418 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":262,"threads_started":1}
I20260619 01:04:09.392979 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:09.392229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:54553)
0619 01:04:09.392674 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:09.392784 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260619 01:04:09.828795 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:09.828081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:44689)
0619 01:04:09.828525 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:09.828605 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260619 01:04:09.946215 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:09.945499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:52159)
0619 01:04:09.945973 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:09.946047 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":255,"threads_started":1}
I20260619 01:04:10.394447 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:10.393815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:34313)
0619 01:04:10.394201 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:10.394275 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":200,"threads_started":1}
I20260619 01:04:10.830430 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:10.829739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:45285)
0619 01:04:10.830168 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:10.830243 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260619 01:04:10.947933 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:10.947224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:49413)
0619 01:04:10.947656 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:10.947734 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":223,"threads_started":1}
I20260619 01:04:11.396193 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:11.395501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:47673)
0619 01:04:11.395937 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:11.396019 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":216,"threads_started":1}
I20260619 01:04:11.824947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16979
I20260619 01:04:11.832393 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:11.831609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:45981)
0619 01:04:11.832082 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:11.832171 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":265,"threads_started":1}
I20260619 01:04:11.949847 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:11.948978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.1:34353)
0619 01:04:11.949526 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:11.949644 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
I20260619 01:04:12.397842 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:12.397150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.2:52127)
0619 01:04:12.397603 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:12.397691 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":223,"threads_started":1}
I20260619 01:04:12.786458 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:11.826327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.0.97.1:35395)
0619 01:04:11.826941 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:11.826957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:12.771520 (+944563us) server_negotiation.cc:407] Connection header received
0619 01:04:12.773094 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:12.773103 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:12.773541 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:12.773834 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:12.775303 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.776495 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.777710 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.778359 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.778485 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:12.781033 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:12.781040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:12.781055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:12.781097 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:12.783723 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.784353 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.784358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.784364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.784523 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.784830 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.784833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.784835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.785316 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:12.785525 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:12.785803 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 01:04:12.786164 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":222,"threads_started":1}
I20260619 01:04:12.787820 17142 generic_service.cc:196] Checking for leaks (request via RPC)
W20260619 01:04:12.827647 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 16979: Timed out: CheckLeaks RPC to 127.0.97.1:35395 timed out after 1.000s (SENT)
I20260619 01:04:12.827896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 16979
I20260619 01:04:12.839310 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:12.833354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35037 (local address 127.0.97.3:42029)
0619 01:04:12.839083 (+ 5729us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:12.839171 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:35037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5626,"thread_start_us":5503,"threads_started":1}
I20260619 01:04:12.846997 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17150
I20260619 01:04:12.863684 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:12.847782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.97.2:44651)
0619 01:04:12.848367 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:12.848404 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:12.848447 (+ 43us) server_negotiation.cc:407] Connection header received
0619 01:04:12.850010 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:12.850019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:12.850497 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:12.850872 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:12.852091 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.853293 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.854533 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.855345 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.855484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:12.858002 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:12.858009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:12.858023 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:12.858059 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:12.860860 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.861466 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.861471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.861477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.861640 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.861938 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.861941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.861943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.862413 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:12.862618 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:12.863064 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 01:04:12.863365 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":268,"threads_started":1}
I20260619 01:04:12.864977 17273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:12.947485 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17150
I20260619 01:04:12.965128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17281
I20260619 01:04:12.982949 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:12.966152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.0.97.3:37485)
0619 01:04:12.966803 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:12.966817 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:12.966873 (+ 56us) server_negotiation.cc:407] Connection header received
0619 01:04:12.968417 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:12.968425 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:12.969024 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:12.969563 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:12.971555 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.972893 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.974035 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:12.974692 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:12.974890 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:12.977218 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:12.977225 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:12.977240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:12.977274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:12.980031 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.980720 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.980726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.980732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.980883 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:12.981202 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:12.981205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:12.981207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:12.981708 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:12.981909 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:12.982261 (+ 352us) server_negotiation.cc:299] Negotiation successful
0619 01:04:12.982587 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":280,"threads_started":1}
I20260619 01:04:12.984268 17404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:13.065678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17281
2026-06-19T01:04:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64960 ms)
[----------] 30 tests from SecurityITest (292839 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:13 dist-test-slave-3rxd krb5kdc[17469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:13 dist-test-slave-3rxd krb5kdc[17469](info): set up 2 sockets
Jun 19 01:04:13 dist-test-slave-3rxd krb5kdc[17469](info): commencing operation
krb5kdc: starting...
W20260619 01:04:14.150815 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:14 dist-test-slave-3rxd krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:04:14.374354 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40721
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:14.687410 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:14.687724 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:14.687922 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:14.694988 17482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:14.695116 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:14.695219 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:14.695323 17482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:14.695415 17482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:14.704810 17482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:14.706044 17482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:14.707336 17482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:14.718946 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:14.718909 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:14.719444 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:14.719784 17482 server_base.cc:1034] running on GCE node
I20260619 01:04:14.720567 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:14.721904 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:14.723093 17482 hybrid_clock.cc:648] HybridClock initialized: now 1781831054723030 us; error 55 us; skew 500 ppm
I20260619 01:04:14.723435 17482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:14 dist-test-slave-3rxd krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:14.747601 17482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:04:14.749095 17482 webserver.cc:460] Webserver started at http://127.0.97.62:37921/ using document root <none> and password file <none>
I20260619 01:04:14.749755 17482 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:14.749850 17482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:14.750094 17482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:14.753067 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "086380b865a6466998494efb19788a49"
format_stamp: "Formatted at 2026-06-19 01:04:14 on dist-test-slave-3rxd"
server_key: "81cc3ef850c0130c3ce5877d00b81a30"
server_key_iv: "bb44e2906cef3291e85f857d93ce035c"
server_key_version: "encryptionkey@0"
I20260619 01:04:14.753733 17482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "086380b865a6466998494efb19788a49"
format_stamp: "Formatted at 2026-06-19 01:04:14 on dist-test-slave-3rxd"
server_key: "81cc3ef850c0130c3ce5877d00b81a30"
server_key_iv: "bb44e2906cef3291e85f857d93ce035c"
server_key_version: "encryptionkey@0"
I20260619 01:04:14.758689 17482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 01:04:14.761880 17497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:14.763173 17482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:04:14.763348 17482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "086380b865a6466998494efb19788a49"
format_stamp: "Formatted at 2026-06-19 01:04:14 on dist-test-slave-3rxd"
server_key: "81cc3ef850c0130c3ce5877d00b81a30"
server_key_iv: "bb44e2906cef3291e85f857d93ce035c"
server_key_version: "encryptionkey@0"
I20260619 01:04:14.763533 17482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:14.781594 17482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:14.785121 17482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:14.785399 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:14.803357 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40721
I20260619 01:04:14.803442 17548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40721 every 8 connection(s)
I20260619 01:04:14.804678 17482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:14.808360 17549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:14.815032 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17482
I20260619 01:04:14.815270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:14.816150 388 external_mini_cluster.cc:1428] Setting key abe614d27aea392616cfad572a92301a
I20260619 01:04:14.819443 17549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: Bootstrap starting.
I20260619 01:04:14.823155 17549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:14.824316 17549 log.cc:824] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:14.827479 17549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: No bootstrap required, opened a new log
Jun 19 01:04:14 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:14.835649 17549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } }
I20260619 01:04:14.836061 17549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:14.836217 17549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086380b865a6466998494efb19788a49, State: Initialized, Role: FOLLOWER
I20260619 01:04:14.836898 17549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } }
I20260619 01:04:14.837177 17549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:14.837337 17549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:14.837520 17549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:14.839547 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:14.818084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.0.97.62:40721)
0619 01:04:14.818890 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:14.818902 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:14.818942 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:04:14.820788 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:14.820812 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:14.821193 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:14.821528 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:14.823547 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:14.825038 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:14.826895 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:14.827530 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:14.827723 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:14.831932 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:14.831940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:14.831954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:14.831999 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:14.835358 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:14.836056 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:14.836063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:14.836069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:14.836276 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:14.836664 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:14.836668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:14.836672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:14.837253 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:14.837577 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:14.837967 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 01:04:14.838412 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":379,"threads_started":1}
I20260619 01:04:14.840229 17549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } }
I20260619 01:04:14.840777 17549 leader_election.cc:302] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 086380b865a6466998494efb19788a49; no voters:
I20260619 01:04:14.841420 17549 leader_election.cc:288] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:14.841602 17554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:14.842286 17554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [term 1 LEADER]: Becoming Leader. State: Replica: 086380b865a6466998494efb19788a49, State: Running, Role: LEADER
I20260619 01:04:14.842969 17554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } }
I20260619 01:04:14.844555 17549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:14.845957 17555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 086380b865a6466998494efb19788a49. Latest consensus state: current_term: 1 leader_uuid: "086380b865a6466998494efb19788a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } } }
I20260619 01:04:14.846622 17555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:14.847824 17556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "086380b865a6466998494efb19788a49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "086380b865a6466998494efb19788a49" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40721 } } }
I20260619 01:04:14.848191 17556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:14.848905 17561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:14.855530 17561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:14.865698 17561 catalog_manager.cc:1347] Generated new cluster ID: 644c2a5c293b43568cb231ccb5723ec1
I20260619 01:04:14.865823 17561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:14.900425 17561 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:14.902343 17561 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:14.921502 17561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: Generated new TSK 0
I20260619 01:04:14.922309 17561 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:04:14.989961 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:15.312917 17575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:15.313225 17575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:15.313401 17575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:15.320590 17575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:15.320765 17575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:15.320955 17575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:15.330289 17575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:15.331619 17575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:15.332902 17575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:15.345746 17580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:15.345697 17581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:15.345772 17583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:15.346578 17575 server_base.cc:1034] running on GCE node
I20260619 01:04:15.347146 17575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:15.348037 17575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:15.349252 17575 hybrid_clock.cc:648] HybridClock initialized: now 1781831055349216 us; error 47 us; skew 500 ppm
I20260619 01:04:15.349560 17575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:15 dist-test-slave-3rxd krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:15.352653 17575 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:04:15.354027 17575 webserver.cc:460] Webserver started at http://127.0.97.1:36649/ using document root <none> and password file <none>
I20260619 01:04:15.354672 17575 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:15.354837 17575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:15.355134 17575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:15.358074 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "31b8ebcd7eeb470b947513d30ad68a89"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "b93b184df2b14e5cb637edfb629d03fc"
server_key_iv: "64e472e583a94c00e0eff3eaeb3a359c"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.358959 17575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31b8ebcd7eeb470b947513d30ad68a89"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "b93b184df2b14e5cb637edfb629d03fc"
server_key_iv: "64e472e583a94c00e0eff3eaeb3a359c"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.364429 17575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 01:04:15.367753 17590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.369025 17575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:04:15.369190 17575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "31b8ebcd7eeb470b947513d30ad68a89"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "b93b184df2b14e5cb637edfb629d03fc"
server_key_iv: "64e472e583a94c00e0eff3eaeb3a359c"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.369392 17575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:15.380214 17575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:15.384223 17575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:15.384500 17575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:15.385397 17575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:15.386620 17575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:15.386742 17575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.386883 17575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:15.386953 17575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.418603 17575 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37719
I20260619 01:04:15.418649 17702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37719 every 8 connection(s)
I20260619 01:04:15.419837 17575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:15.425766 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17575
I20260619 01:04:15.425978 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:15.426563 388 external_mini_cluster.cc:1428] Setting key 93113267d89b64769c1dc7d148b729d6
Jun 19 01:04:15 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:15.439960 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:15.422346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47563 (local address 127.0.97.62:40721)
0619 01:04:15.422799 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:15.422803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:15.423760 (+ 957us) server_negotiation.cc:407] Connection header received
0619 01:04:15.425529 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:15.425535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:15.425623 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:15.425743 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:15.427298 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.428253 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.429695 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.430105 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.430236 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:15.433790 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:15.433795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:15.433799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:15.433835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:15.437049 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:15.437730 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:15.437736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:15.437739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:15.437862 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:15.438280 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:15.438284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:15.438288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:15.438551 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:15.438703 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:15.439412 (+ 709us) server_negotiation.cc:299] Negotiation successful
0619 01:04:15.439619 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":197,"threads_started":1}
I20260619 01:04:15.440874 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:15.422935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40721 (local address 127.0.97.1:47563)
0619 01:04:15.423612 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:15.423646 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:15.425276 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:15.426018 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:15.426025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:15.426533 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:15.427141 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:15.427157 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.428416 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.428420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:15.429557 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:15.429572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.430318 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.430326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:15.430562 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:15.431275 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:15.431296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:15.433587 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:15.437178 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:15.437186 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:15.437201 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:15.437605 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:15.437990 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:15.437994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:15.437998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:15.438165 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:15.438673 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:15.438680 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:15.439044 (+ 364us) client_negotiation.cc:769] Sending connection context
0619 01:04:15.439306 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 01:04:15.439775 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":247,"threads_started":1}
I20260619 01:04:15.442482 17703 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40721
I20260619 01:04:15.442836 17703 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:15.443511 17703 heartbeater.cc:511] Master 127.0.97.62:40721 requested a full tablet report, sending...
I20260619 01:04:15.445652 17514 ts_manager.cc:194] Registered new tserver with Master: 31b8ebcd7eeb470b947513d30ad68a89 (127.0.97.1:37719)
I20260619 01:04:15.447113 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47563
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:04:15.477520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:15.794582 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:15.794857 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:15.794960 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:15.801750 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:15.801844 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:15.802000 17709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:15.811689 17709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:15.812791 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:15.813987 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:15.826928 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:15.827133 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:15.826900 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:15.828506 17709 server_base.cc:1034] running on GCE node
I20260619 01:04:15.829031 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:15.829907 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:15.831092 17709 hybrid_clock.cc:648] HybridClock initialized: now 1781831055831033 us; error 50 us; skew 500 ppm
I20260619 01:04:15.831347 17709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:15 dist-test-slave-3rxd krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:15.834362 17709 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:04:15.835788 17709 webserver.cc:460] Webserver started at http://127.0.97.2:35463/ using document root <none> and password file <none>
I20260619 01:04:15.836443 17709 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:15.836570 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:15.836887 17709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:15.839712 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "0e068c0fa3ba4458ba3beb73771c10bc"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "aef49706151de99bc7b21a751b38afc4"
server_key_iv: "d34ab934d17595386d40c7beac9bda7a"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.840430 17709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0e068c0fa3ba4458ba3beb73771c10bc"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "aef49706151de99bc7b21a751b38afc4"
server_key_iv: "d34ab934d17595386d40c7beac9bda7a"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.845129 17709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:15.848402 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.849625 17709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:04:15.849790 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "0e068c0fa3ba4458ba3beb73771c10bc"
format_stamp: "Formatted at 2026-06-19 01:04:15 on dist-test-slave-3rxd"
server_key: "aef49706151de99bc7b21a751b38afc4"
server_key_iv: "d34ab934d17595386d40c7beac9bda7a"
server_key_version: "encryptionkey@0"
I20260619 01:04:15.849990 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:15.866816 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:15.871295 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:15.871578 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:15.872587 17709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:15.873872 17709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:15.873978 17709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.874132 17709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:15.874202 17709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:15.906740 17709 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44383
I20260619 01:04:15.906761 17836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44383 every 8 connection(s)
I20260619 01:04:15.908051 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:15.908304 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17709
I20260619 01:04:15.908473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:15.908993 388 external_mini_cluster.cc:1428] Setting key 84debd2c3f37c3b1ed98305f311285ee
Jun 19 01:04:15 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:15.931486 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:15.910782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57391 (local address 127.0.97.62:40721)
0619 01:04:15.910995 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:15.911003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:15.912325 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 01:04:15.915125 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:15.915131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:15.915214 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:15.915311 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:15.917320 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.918020 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.920445 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.920847 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.920940 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:15.924776 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:15.924785 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:15.924792 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:15.924836 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:15.928281 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:15.928973 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:15.928981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:15.928987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:15.929139 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:15.929564 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:15.929571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:15.929577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:15.929943 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:15.930090 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:15.930755 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 01:04:15.931032 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:04:15.932065 17839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:15.911353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40721 (local address 127.0.97.2:57391)
0619 01:04:15.912160 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:15.912206 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:15.914834 (+ 2628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:15.915576 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:15.915589 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:15.916345 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:15.917128 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:15.917153 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.918179 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.918187 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:15.920312 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:15.920329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:15.921066 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:15.921075 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:15.921383 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:15.922157 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:15.922179 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:15.924560 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:15.928424 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:15.928430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:15.928442 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:15.928825 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:15.929236 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:15.929239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:15.929241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:15.929422 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:15.930054 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:15.930060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:15.930338 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 01:04:15.930570 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 01:04:15.931003 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":313,"threads_started":1}
I20260619 01:04:15.933485 17837 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40721
I20260619 01:04:15.933800 17837 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:15.934804 17837 heartbeater.cc:511] Master 127.0.97.62:40721 requested a full tablet report, sending...
I20260619 01:04:15.936481 17514 ts_manager.cc:194] Registered new tserver with Master: 0e068c0fa3ba4458ba3beb73771c10bc (127.0.97.2:44383)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260619 01:04:15.937727 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:04:15.966071 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:16.302881 17842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:16.303129 17842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:16.303308 17842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:16.310111 17842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:16.310200 17842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:16.310417 17842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:16.319777 17842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:16.320909 17842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:16.322129 17842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:16.335422 17847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:16.335359 17850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:16.335399 17848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:16.336194 17842 server_base.cc:1034] running on GCE node
I20260619 01:04:16.336715 17842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:16.337536 17842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:16.338807 17842 hybrid_clock.cc:648] HybridClock initialized: now 1781831056338749 us; error 55 us; skew 500 ppm
I20260619 01:04:16.339129 17842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:16 dist-test-slave-3rxd krb5kdc[17469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:16.342224 17842 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:04:16.343691 17842 webserver.cc:460] Webserver started at http://127.0.97.3:34833/ using document root <none> and password file <none>
I20260619 01:04:16.344386 17842 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:16.344532 17842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:16.344836 17842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:16.347685 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "ee2acd13933f46dd8629b385dcf88d80"
format_stamp: "Formatted at 2026-06-19 01:04:16 on dist-test-slave-3rxd"
server_key: "cc85505caee4d0df788796824edf634c"
server_key_iv: "904312d4828db2c6d866f26311790672"
server_key_version: "encryptionkey@0"
I20260619 01:04:16.348474 17842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ee2acd13933f46dd8629b385dcf88d80"
format_stamp: "Formatted at 2026-06-19 01:04:16 on dist-test-slave-3rxd"
server_key: "cc85505caee4d0df788796824edf634c"
server_key_iv: "904312d4828db2c6d866f26311790672"
server_key_version: "encryptionkey@0"
I20260619 01:04:16.353349 17842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:04:16.356856 17857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:16.358150 17842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:04:16.358321 17842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "ee2acd13933f46dd8629b385dcf88d80"
format_stamp: "Formatted at 2026-06-19 01:04:16 on dist-test-slave-3rxd"
server_key: "cc85505caee4d0df788796824edf634c"
server_key_iv: "904312d4828db2c6d866f26311790672"
server_key_version: "encryptionkey@0"
I20260619 01:04:16.358513 17842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:16.373302 17842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:16.377009 17842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:16.377285 17842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:16.378194 17842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:16.379451 17842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:16.379556 17842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:16.379711 17842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:16.379783 17842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:16.412393 17842 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38587
I20260619 01:04:16.412477 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38587 every 8 connection(s)
I20260619 01:04:16.413650 17842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:16.418569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 17842
I20260619 01:04:16.418782 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:16.419360 388 external_mini_cluster.cc:1428] Setting key e6af7a7684cefaf552adbca864f54966
Jun 19 01:04:16 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:16.433492 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.416088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32843 (local address 127.0.97.62:40721)
0619 01:04:16.416303 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.416309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.417459 (+ 1150us) server_negotiation.cc:407] Connection header received
0619 01:04:16.419608 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.419613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.419681 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.419740 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:16.421898 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.423161 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.424661 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.425041 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.425136 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.428382 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:16.428393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:16.428396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:16.428426 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:16.430848 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.431452 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.431455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.431457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.431568 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.431914 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.431917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.431919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.432158 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:16.432262 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.432964 (+ 702us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.433111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:04:16.434562 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.416570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40721 (local address 127.0.97.3:32843)
0619 01:04:16.417257 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.417344 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.419279 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.420228 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.420236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.420929 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:16.421668 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.421694 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.423331 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.423335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.424533 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.424545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.425214 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.425218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.425402 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.426057 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:16.426079 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:16.428202 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:16.430962 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:16.430968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:16.430978 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:16.431355 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:16.431658 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:16.431661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:16.431663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:16.431826 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:16.432257 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:16.432262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:16.432592 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.432915 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.433383 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":278,"threads_started":1}
I20260619 01:04:16.435850 17970 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40721
I20260619 01:04:16.436121 17970 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:16.436797 17970 heartbeater.cc:511] Master 127.0.97.62:40721 requested a full tablet report, sending...
I20260619 01:04:16.438083 17514 ts_manager.cc:194] Registered new tserver with Master: ee2acd13933f46dd8629b385dcf88d80 (127.0.97.3:38587)
I20260619 01:04:16.438853 17514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32843
I20260619 01:04:16.447041 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:16.449726 17703 heartbeater.cc:503] Master 127.0.97.62:40721 was elected leader, sending a full tablet report...
I20260619 01:04:16.464183 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.451636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45024 (local address 127.0.97.62:40721)
0619 01:04:16.451792 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.451796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.452239 (+ 443us) server_negotiation.cc:407] Connection header received
0619 01:04:16.452367 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.452371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.452428 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.452505 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:16.453987 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.454777 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.456093 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.456534 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.456678 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.459216 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:16.459222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:16.459229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:16.459263 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:16.461540 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.462176 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.462181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.462185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.462296 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.462690 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.462694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.462697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.463031 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:16.463299 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.463640 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.463846 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:04:16.468066 17514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:16.474823 17514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:16.508642 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.497768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.0.97.2:44383)
0619 01:04:16.501479 (+ 3711us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.501485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.501506 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:04:16.501588 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.501593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.501729 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.501922 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:16.502466 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503768 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.506257 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507380 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.507533 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.507720 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.508149 (+ 429us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.508320 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3561,"thread_start_us":3440,"threads_started":1}
I20260619 01:04:16.508664 17984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.486151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37719 (local address 127.0.0.1:34198)
0619 01:04:16.496070 (+ 9919us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.496120 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.496341 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.500756 (+ 4415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.500762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.500784 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:16.501272 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.501286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503283 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.503289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.505986 (+ 2697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.506003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507802 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.507810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.507960 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.507992 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.508058 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.508219 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9592,"mutex_wait_us":26,"thread_start_us":14721,"threads_started":1}
I20260619 01:04:16.508805 17982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.486157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38587 (local address 127.0.0.1:54738)
0619 01:04:16.498004 (+ 11847us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.498025 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.498150 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.500138 (+ 1988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.500151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.500196 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:16.500678 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.500694 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503283 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.503289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.506256 (+ 2967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.506269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507993 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.508002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.508178 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.508207 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.508291 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.508422 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11714,"mutex_wait_us":575,"thread_start_us":10154,"threads_started":1}
I20260619 01:04:16.508841 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.486662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54738 (local address 127.0.97.3:38587)
0619 01:04:16.499081 (+ 12419us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.499094 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.499136 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:04:16.499256 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.499267 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.499642 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.500646 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:16.500981 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503127 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.506411 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507730 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.507846 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.508068 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.508360 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.508503 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12187,"thread_start_us":13117,"threads_started":1}
I20260619 01:04:16.509439 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.497484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44383 (local address 127.0.0.1:37236)
0619 01:04:16.499554 (+ 2070us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.499570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.500459 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.501908 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.501912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.501936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:16.502280 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.502292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503911 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.503919 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.506083 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.506100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507560 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.507566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.507747 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.507827 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.508970 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.509095 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1924,"mutex_wait_us":4,"thread_start_us":1785,"threads_started":1}
I20260619 01:04:16.509891 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.486249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.97.1:37719)
0619 01:04:16.500081 (+ 13832us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.500089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.500115 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:04:16.500317 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.500324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.500559 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.501492 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:16.501691 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.503159 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.506161 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.507576 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.509139 (+ 1563us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.509371 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.509433 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.509619 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13649,"thread_start_us":13960,"threads_started":1}
I20260619 01:04:16.512636 17905 tablet_service.cc:1467] Processing CreateTablet for tablet bd51016574564bf8a3bd875653c97a3c (DEFAULT_TABLE table=test-table [id=be92f3fdc7274a87aedbbc40bc16a616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:16.513406 17772 tablet_service.cc:1467] Processing CreateTablet for tablet bd51016574564bf8a3bd875653c97a3c (DEFAULT_TABLE table=test-table [id=be92f3fdc7274a87aedbbc40bc16a616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:16.513890 17905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd51016574564bf8a3bd875653c97a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:16.513389 17638 tablet_service.cc:1467] Processing CreateTablet for tablet bd51016574564bf8a3bd875653c97a3c (DEFAULT_TABLE table=test-table [id=be92f3fdc7274a87aedbbc40bc16a616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:16.514710 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd51016574564bf8a3bd875653c97a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:16.514710 17638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd51016574564bf8a3bd875653c97a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:16.525377 17988 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Bootstrap starting.
I20260619 01:04:16.529410 17988 tablet_bootstrap.cc:654] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:16.530018 17990 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Bootstrap starting.
I20260619 01:04:16.531033 17988 log.cc:824] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:16.531504 17989 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Bootstrap starting.
I20260619 01:04:16.533653 17990 tablet_bootstrap.cc:654] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:16.534922 17990 log.cc:824] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:16.535245 17989 tablet_bootstrap.cc:654] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:16.536399 17989 log.cc:824] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:16.536619 17988 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: No bootstrap required, opened a new log
I20260619 01:04:16.536885 17988 ts_tablet_manager.cc:1397] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260619 01:04:16.537550 17990 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: No bootstrap required, opened a new log
I20260619 01:04:16.537806 17990 ts_tablet_manager.cc:1397] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 01:04:16.544694 17989 tablet_bootstrap.cc:492] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: No bootstrap required, opened a new log
I20260619 01:04:16.544991 17989 ts_tablet_manager.cc:1397] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260619 01:04:16.545591 17990 raft_consensus.cc:348] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.546069 17990 raft_consensus.cc:374] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:16.546187 17990 raft_consensus.cc:729] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8ebcd7eeb470b947513d30ad68a89, State: Initialized, Role: FOLLOWER
I20260619 01:04:16.546859 17990 consensus_queue.cc:260] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.548524 17990 ts_tablet_manager.cc:1428] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260619 01:04:16.548547 17988 raft_consensus.cc:348] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.548960 17988 raft_consensus.cc:374] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:16.549064 17988 raft_consensus.cc:729] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2acd13933f46dd8629b385dcf88d80, State: Initialized, Role: FOLLOWER
I20260619 01:04:16.549682 17988 consensus_queue.cc:260] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.550812 17970 heartbeater.cc:503] Master 127.0.97.62:40721 was elected leader, sending a full tablet report...
I20260619 01:04:16.551357 17988 ts_tablet_manager.cc:1428] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260619 01:04:16.555593 17989 raft_consensus.cc:348] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.555984 17989 raft_consensus.cc:374] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:16.556088 17989 raft_consensus.cc:729] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e068c0fa3ba4458ba3beb73771c10bc, State: Initialized, Role: FOLLOWER
I20260619 01:04:16.556641 17989 consensus_queue.cc:260] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.557654 17837 heartbeater.cc:503] Master 127.0.97.62:40721 was elected leader, sending a full tablet report...
I20260619 01:04:16.558239 17989 ts_tablet_manager.cc:1428] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260619 01:04:16.660240 17838 tablet.cc:2369] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:04:16.665305 17971 tablet.cc:2369] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:04:16.672068 17704 tablet.cc:2369] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:16.714323 17996 raft_consensus.cc:482] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:04:16.714669 17996 raft_consensus.cc:504] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.716457 17996 leader_election.cc:288] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b8ebcd7eeb470b947513d30ad68a89 (127.0.97.1:37719), ee2acd13933f46dd8629b385dcf88d80 (127.0.97.3:38587)
I20260619 01:04:16.725251 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.717401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44075 (local address 127.0.97.1:37719)
0619 01:04:16.717634 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.717639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.717760 (+ 121us) server_negotiation.cc:407] Connection header received
0619 01:04:16.718087 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.718096 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.718222 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.718316 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:16.718953 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.720030 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.722911 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.724112 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.724248 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.724348 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.724862 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.724990 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:04:16.725247 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.716928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52817 (local address 127.0.97.3:38587)
0619 01:04:16.717199 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.717207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.718030 (+ 823us) server_negotiation.cc:407] Connection header received
0619 01:04:16.718285 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.718292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.718366 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.718450 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:16.719127 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.720139 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.722917 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.724131 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.724248 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.724348 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.724862 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.724997 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:04:16.725515 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.716843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38587 (local address 127.0.97.2:52817)
0619 01:04:16.717919 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.717941 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.718169 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.718487 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.718491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.718527 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:16.718985 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.718995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.720266 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.720272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.722740 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.722756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.724391 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.724398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.724672 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.724725 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.724822 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.725026 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":140,"thread_start_us":743,"threads_started":1}
I20260619 01:04:16.725759 17997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.716729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37719 (local address 127.0.97.2:44075)
0619 01:04:16.717624 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:16.717649 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:16.717829 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:16.718347 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:16.718350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:16.718370 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:16.718803 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.718818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.720153 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.720156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.722742 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:16.722756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.724401 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.724405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:16.724678 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.724728 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 01:04:16.724821 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 01:04:16.725027 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":297,"thread_start_us":520,"threads_started":1}
I20260619 01:04:16.726616 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd51016574564bf8a3bd875653c97a3c" candidate_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2acd13933f46dd8629b385dcf88d80" is_pre_election: true
I20260619 01:04:16.726657 17658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd51016574564bf8a3bd875653c97a3c" candidate_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8ebcd7eeb470b947513d30ad68a89" is_pre_election: true
I20260619 01:04:16.727195 17925 raft_consensus.cc:2454] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e068c0fa3ba4458ba3beb73771c10bc in term 0.
I20260619 01:04:16.727210 17658 raft_consensus.cc:2454] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e068c0fa3ba4458ba3beb73771c10bc in term 0.
I20260619 01:04:16.727844 17728 leader_election.cc:302] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e068c0fa3ba4458ba3beb73771c10bc, 31b8ebcd7eeb470b947513d30ad68a89; no voters:
I20260619 01:04:16.728233 17996 raft_consensus.cc:2781] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:16.728395 17996 raft_consensus.cc:482] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:16.728502 17996 raft_consensus.cc:3037] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:16.730659 17996 raft_consensus.cc:504] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.731343 17996 leader_election.cc:288] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [CANDIDATE]: Term 1 election: Requested vote from peers 31b8ebcd7eeb470b947513d30ad68a89 (127.0.97.1:37719), ee2acd13933f46dd8629b385dcf88d80 (127.0.97.3:38587)
I20260619 01:04:16.731714 17658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd51016574564bf8a3bd875653c97a3c" candidate_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b8ebcd7eeb470b947513d30ad68a89"
I20260619 01:04:16.731786 17925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd51016574564bf8a3bd875653c97a3c" candidate_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2acd13933f46dd8629b385dcf88d80"
I20260619 01:04:16.731928 17658 raft_consensus.cc:3037] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:16.732028 17925 raft_consensus.cc:3037] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:16.734472 17658 raft_consensus.cc:2454] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e068c0fa3ba4458ba3beb73771c10bc in term 1.
I20260619 01:04:16.734647 17925 raft_consensus.cc:2454] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e068c0fa3ba4458ba3beb73771c10bc in term 1.
I20260619 01:04:16.735009 17728 leader_election.cc:302] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e068c0fa3ba4458ba3beb73771c10bc, 31b8ebcd7eeb470b947513d30ad68a89; no voters:
I20260619 01:04:16.735306 17996 raft_consensus.cc:2781] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:16.735757 17996 raft_consensus.cc:686] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 1 LEADER]: Becoming Leader. State: Replica: 0e068c0fa3ba4458ba3beb73771c10bc, State: Running, Role: LEADER
I20260619 01:04:16.736227 17996 consensus_queue.cc:237] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } }
I20260619 01:04:16.740797 17513 catalog_manager.cc:5495] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e068c0fa3ba4458ba3beb73771c10bc (127.0.97.2). New cstate: current_term: 1 leader_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e068c0fa3ba4458ba3beb73771c10bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44383 } health_report { overall_health: HEALTHY } } }
I20260619 01:04:16.804008 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.798515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.97.2:44383)
0619 01:04:16.798757 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.798763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.798823 (+ 60us) server_negotiation.cc:407] Connection header received
0619 01:04:16.799095 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.799100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.799184 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.799333 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:16.799715 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.800434 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.801894 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.802292 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.802424 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.802938 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:16.803438 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:16.803614 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.803661 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.803739 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:04:16.812996 17658 raft_consensus.cc:1261] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 1 FOLLOWER]: Refusing update from remote peer 0e068c0fa3ba4458ba3beb73771c10bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:16.812997 17925 raft_consensus.cc:1261] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 1 FOLLOWER]: Refusing update from remote peer 0e068c0fa3ba4458ba3beb73771c10bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:16.814050 17999 consensus_queue.cc:1035] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2acd13933f46dd8629b385dcf88d80" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:16.814399 17996 consensus_queue.cc:1035] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b8ebcd7eeb470b947513d30ad68a89" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:16.826231 18002 mvcc.cc:205] Tried to move back new op lower bound from 7298380008695357440 to 7298380008395075584. Current Snapshot: MvccSnapshot[applied={T|T < 7298380008695357440}]
I20260619 01:04:16.830288 18004 mvcc.cc:205] Tried to move back new op lower bound from 7298380008695357440 to 7298380008395075584. Current Snapshot: MvccSnapshot[applied={T|T < 7298380008695357440}]
I20260619 01:04:16.838891 17513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:16.839143 17513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:16.842464 17513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 086380b865a6466998494efb19788a49: Sending DeleteTablet for 3 replicas of tablet bd51016574564bf8a3bd875653c97a3c
I20260619 01:04:16.843713 17638 tablet_service.cc:1514] Processing DeleteTablet for tablet bd51016574564bf8a3bd875653c97a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34198
I20260619 01:04:16.843992 17772 tablet_service.cc:1514] Processing DeleteTablet for tablet bd51016574564bf8a3bd875653c97a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37236
I20260619 01:04:16.844645 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17575
I20260619 01:04:16.845643 17905 tablet_service.cc:1514] Processing DeleteTablet for tablet bd51016574564bf8a3bd875653c97a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:16 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54738
I20260619 01:04:16.858144 18012 tablet_replica.cc:330] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: stopping tablet replica
I20260619 01:04:16.858701 18012 raft_consensus.cc:2229] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:16.859496 18012 raft_consensus.cc:2258] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:16.869772 18012 ts_tablet_manager.cc:1905] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:16.870411 18013 tablet_replica.cc:330] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: stopping tablet replica
I20260619 01:04:16.870977 18013 raft_consensus.cc:2229] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:16.870698 18011 tablet_replica.cc:330] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: stopping tablet replica
I20260619 01:04:16.871424 18011 raft_consensus.cc:2229] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:16.871623 18013 raft_consensus.cc:2258] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:16.871897 18011 raft_consensus.cc:2258] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:16.873735 18013 ts_tablet_manager.cc:1905] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:16.874272 18011 ts_tablet_manager.cc:1905] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:16.879554 18012 ts_tablet_manager.cc:1918] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:16.879778 18012 log.cc:1196] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/bd51016574564bf8a3bd875653c97a3c
I20260619 01:04:16.880398 18012 ts_tablet_manager.cc:1939] T bd51016574564bf8a3bd875653c97a3c P 0e068c0fa3ba4458ba3beb73771c10bc: Deleting consensus metadata
I20260619 01:04:16.880955 18011 ts_tablet_manager.cc:1918] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:16.881104 18011 log.cc:1196] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/bd51016574564bf8a3bd875653c97a3c
I20260619 01:04:16.881626 18011 ts_tablet_manager.cc:1939] T bd51016574564bf8a3bd875653c97a3c P 31b8ebcd7eeb470b947513d30ad68a89: Deleting consensus metadata
Jun 19 01:04:16 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:04:16.882572 17498 catalog_manager.cc:4841] TS 0e068c0fa3ba4458ba3beb73771c10bc (127.0.97.2:44383): tablet bd51016574564bf8a3bd875653c97a3c (table test-table [id=be92f3fdc7274a87aedbbc40bc16a616]) successfully deleted
I20260619 01:04:16.883251 17501 catalog_manager.cc:4841] TS 31b8ebcd7eeb470b947513d30ad68a89 (127.0.97.1:37719): tablet bd51016574564bf8a3bd875653c97a3c (table test-table [id=be92f3fdc7274a87aedbbc40bc16a616]) successfully deleted
I20260619 01:04:16.884744 18013 ts_tablet_manager.cc:1918] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:16.884935 18013 log.cc:1196] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/bd51016574564bf8a3bd875653c97a3c
I20260619 01:04:16.885452 18013 ts_tablet_manager.cc:1939] T bd51016574564bf8a3bd875653c97a3c P ee2acd13933f46dd8629b385dcf88d80: Deleting consensus metadata
I20260619 01:04:16.886780 17498 catalog_manager.cc:4841] TS ee2acd13933f46dd8629b385dcf88d80 (127.0.97.3:38587): tablet bd51016574564bf8a3bd875653c97a3c (table test-table [id=be92f3fdc7274a87aedbbc40bc16a616]) successfully deleted
I20260619 01:04:16.889696 17983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:16.845993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.0.97.1:37719)
0619 01:04:16.846209 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:16.846218 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:16.861932 (+ 15714us) server_negotiation.cc:407] Connection header received
0619 01:04:16.862188 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:16.862196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:16.862311 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:16.862436 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:16.868789 (+ 6353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.870363 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.872427 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:16.873112 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:16.873210 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:16.883159 (+ 9949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:16.883175 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:16.883193 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:16.883238 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:16.886975 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.887630 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.887639 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.887642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.887802 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:16.888134 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:16.888140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:16.888143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:16.888608 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:16.888939 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:16.889098 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 01:04:16.889304 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 01:04:16.890357 17698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:16.997066 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17575
I20260619 01:04:17.020390 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17709
Jun 19 01:04:17 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:04:17.038035 17987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:17.021170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.0.97.2:44383)
0619 01:04:17.021349 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:17.021355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:17.021507 (+ 152us) server_negotiation.cc:407] Connection header received
0619 01:04:17.021714 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:17.021729 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:17.021792 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:17.021912 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:17.023407 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.024233 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.025546 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.025979 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.026118 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:17.030368 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:17.030393 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:17.030415 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:17.030464 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:17.034415 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:17.035462 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:17.035473 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:17.035477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:17.035674 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:17.036158 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:17.036166 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:17.036171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:17.036715 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:17.037089 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:17.037354 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 01:04:17.037607 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:04:17.038882 17832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:17.150403 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17709
I20260619 01:04:17.169016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17842
Jun 19 01:04:17 dist-test-slave-3rxd krb5kdc[17469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:04:17.183033 17985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:17.169681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.0.97.3:38587)
0619 01:04:17.169904 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:17.169908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:17.169926 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:04:17.170086 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:17.170090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:17.170154 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:17.170236 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:17.171567 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.172256 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.173413 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.173759 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.173846 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:17.177903 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:17.177916 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:17.177932 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:17.177963 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:17.180497 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:17.181069 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:17.181075 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:17.181077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:17.181202 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:17.181495 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:17.181498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:17.181500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:17.181819 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:17.182023 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:17.182251 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 01:04:17.182425 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:04:17.183634 17965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:17.269994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17842
I20260619 01:04:17.286221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 17482
I20260619 01:04:17.287325 17544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:17.396106 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 17482
2026-06-19T01:04:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4332 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T01:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:17Z Disabled control of system clock
I20260619 01:04:17.469144 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40155
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:17.784647 18022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:17.784914 18022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:17.785073 18022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:17.792162 18022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:17.792258 18022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:17.792294 18022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:17.792367 18022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:17.792436 18022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:17.802053 18022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:17.803390 18022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:17.805127 18022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:17.816984 18027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:17.817030 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:17.817078 18028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:17.820168 18022 server_base.cc:1034] running on GCE node
I20260619 01:04:17.820933 18022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:17.822331 18022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:17.823570 18022 hybrid_clock.cc:648] HybridClock initialized: now 1781831057823538 us; error 44 us; skew 500 ppm
I20260619 01:04:17.823920 18022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:17.826120 18022 webserver.cc:460] Webserver started at http://127.0.97.62:37369/ using document root <none> and password file <none>
I20260619 01:04:17.826865 18022 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:17.827001 18022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:17.827329 18022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:17.830183 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "7331a08ee30a44409fe70020ef9ceaa3"
format_stamp: "Formatted at 2026-06-19 01:04:17 on dist-test-slave-3rxd"
server_key: "28b4d9030505499d031fe6b9badaefd6"
server_key_iv: "8ceeaf0da9000bd3f1938793e6348aba"
server_key_version: "encryptionkey@0"
I20260619 01:04:17.831055 18022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "7331a08ee30a44409fe70020ef9ceaa3"
format_stamp: "Formatted at 2026-06-19 01:04:17 on dist-test-slave-3rxd"
server_key: "28b4d9030505499d031fe6b9badaefd6"
server_key_iv: "8ceeaf0da9000bd3f1938793e6348aba"
server_key_version: "encryptionkey@0"
I20260619 01:04:17.836009 18022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 01:04:17.839246 18036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:17.840525 18022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:04:17.840711 18022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "7331a08ee30a44409fe70020ef9ceaa3"
format_stamp: "Formatted at 2026-06-19 01:04:17 on dist-test-slave-3rxd"
server_key: "28b4d9030505499d031fe6b9badaefd6"
server_key_iv: "8ceeaf0da9000bd3f1938793e6348aba"
server_key_version: "encryptionkey@0"
I20260619 01:04:17.840916 18022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:17.860767 18022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:17.861647 18022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:17.861965 18022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:17.880381 18022 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40155
I20260619 01:04:17.880385 18087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40155 every 8 connection(s)
I20260619 01:04:17.881767 18022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:17.883827 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18022
I20260619 01:04:17.884011 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:17.884578 388 external_mini_cluster.cc:1428] Setting key 029ef3292f2f63b72935cc9390f0c5fc
I20260619 01:04:17.886109 18088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:17.897382 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:17.886449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.0.97.62:40155)
0619 01:04:17.887423 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:17.887434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:17.887473 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:04:17.888979 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:17.888997 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:17.889417 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:17.889826 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:17.890357 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.891749 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.893159 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:17.893775 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:17.893969 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:17.895161 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:17.895171 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:17.895183 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:17.895767 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:17.895839 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:17.895858 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:17.895982 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:17.896190 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 01:04:17.896472 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":354,"threads_started":1}
I20260619 01:04:17.897420 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: Bootstrap starting.
I20260619 01:04:17.900688 18088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:17.901648 18088 log.cc:824] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:17.904528 18088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: No bootstrap required, opened a new log
I20260619 01:04:17.912223 18088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } }
I20260619 01:04:17.912622 18088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:17.912719 18088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7331a08ee30a44409fe70020ef9ceaa3, State: Initialized, Role: FOLLOWER
I20260619 01:04:17.913339 18088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } }
I20260619 01:04:17.913584 18088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:17.913722 18088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:17.913878 18088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:17.916219 18088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } }
I20260619 01:04:17.916824 18088 leader_election.cc:302] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7331a08ee30a44409fe70020ef9ceaa3; no voters:
I20260619 01:04:17.917464 18088 leader_election.cc:288] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:17.917569 18093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:17.918299 18093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [term 1 LEADER]: Becoming Leader. State: Replica: 7331a08ee30a44409fe70020ef9ceaa3, State: Running, Role: LEADER
I20260619 01:04:17.918874 18093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } }
I20260619 01:04:17.919603 18088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:17.921794 18094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7331a08ee30a44409fe70020ef9ceaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } } }
I20260619 01:04:17.921810 18095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7331a08ee30a44409fe70020ef9ceaa3. Latest consensus state: current_term: 1 leader_uuid: "7331a08ee30a44409fe70020ef9ceaa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7331a08ee30a44409fe70020ef9ceaa3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40155 } } }
I20260619 01:04:17.922224 18094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:17.922283 18095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:17.924839 18100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:17.929528 18100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:17.937691 18100 catalog_manager.cc:1347] Generated new cluster ID: 3d69a3a60a1a4b56acab5898d9d9f8e1
I20260619 01:04:17.937827 18100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:17.962374 18100 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:17.963707 18100 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:17.972152 18100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: Generated new TSK 0
I20260619 01:04:17.972945 18100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:17.986819 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:18.311932 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:18.312228 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:18.312390 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:18.319501 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:18.319620 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:18.319824 18112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:18.329135 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:18.330473 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:18.332268 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:18.345118 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:18.345199 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:18.345196 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:18.345793 18112 server_base.cc:1034] running on GCE node
I20260619 01:04:18.346395 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:18.347292 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:18.348505 18112 hybrid_clock.cc:648] HybridClock initialized: now 1781831058348462 us; error 34 us; skew 500 ppm
I20260619 01:04:18.348830 18112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:18.350961 18112 webserver.cc:460] Webserver started at http://127.0.97.1:39007/ using document root <none> and password file <none>
I20260619 01:04:18.351621 18112 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:18.351743 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:18.352057 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:18.354847 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "d5e04e3031764b6d94aaed643ee2605d"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "bb70384e330b28fe9c7a3d5f30ebed01"
server_key_iv: "0767a6e963ab448179748a23ba78aabd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.355628 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d5e04e3031764b6d94aaed643ee2605d"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "bb70384e330b28fe9c7a3d5f30ebed01"
server_key_iv: "0767a6e963ab448179748a23ba78aabd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.360342 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:04:18.363485 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.364674 18112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:04:18.364837 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "d5e04e3031764b6d94aaed643ee2605d"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "bb70384e330b28fe9c7a3d5f30ebed01"
server_key_iv: "0767a6e963ab448179748a23ba78aabd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.365047 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:18.402143 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:18.402999 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:18.403230 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:18.404099 18112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:18.405483 18112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:18.405565 18112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.405640 18112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:18.405694 18112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.438284 18112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37447
I20260619 01:04:18.438364 18238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37447 every 8 connection(s)
I20260619 01:04:18.439605 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:18.442226 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18112
I20260619 01:04:18.442394 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:18.442940 388 external_mini_cluster.cc:1428] Setting key 915a1264192102d4b65017751ac1c72b
I20260619 01:04:18.445860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:18.452292 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:18.442034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39949 (local address 127.0.97.62:40155)
0619 01:04:18.442536 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:18.442542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:18.443972 (+ 1430us) server_negotiation.cc:407] Connection header received
0619 01:04:18.446039 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:18.446045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:18.446141 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:18.446256 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:18.447330 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.448069 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.449585 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.449976 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.450105 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:18.451468 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:18.451475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:18.451480 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:18.451504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:18.451547 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:18.451560 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:18.451654 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:18.451870 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 01:04:18.451973 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":249,"threads_started":1}
I20260619 01:04:18.453212 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:18.442694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40155 (local address 127.0.97.1:39949)
0619 01:04:18.443821 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:18.443856 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:18.445737 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:18.446483 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:18.446491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:18.446577 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:18.447171 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:18.447187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.448228 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.448232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:18.449420 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:18.449450 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.450197 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.450206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:18.450437 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:18.451147 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:18.451150 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:18.451186 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:18.451259 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:18.451654 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:18.451667 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:18.451677 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:04:18.451843 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 01:04:18.452203 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":347,"threads_started":1}
I20260619 01:04:18.454687 18239 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40155
I20260619 01:04:18.454990 18239 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:18.455672 18239 heartbeater.cc:511] Master 127.0.97.62:40155 requested a full tablet report, sending...
I20260619 01:04:18.457736 18053 ts_manager.cc:194] Registered new tserver with Master: d5e04e3031764b6d94aaed643ee2605d (127.0.97.1:37447)
I20260619 01:04:18.459959 18053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:39949
W20260619 01:04:18.768080 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:18.768441 18243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:18.768550 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:18.775840 18243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:18.775949 18243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:18.776069 18243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:18.785000 18243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:18.786326 18243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:18.788197 18243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:18.801643 18248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:18.801666 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:18.801928 18249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:18.802363 18243 server_base.cc:1034] running on GCE node
I20260619 01:04:18.803033 18243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:18.803934 18243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:18.805160 18243 hybrid_clock.cc:648] HybridClock initialized: now 1781831058805108 us; error 48 us; skew 500 ppm
I20260619 01:04:18.805472 18243 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:18.807555 18243 webserver.cc:460] Webserver started at http://127.0.97.2:44203/ using document root <none> and password file <none>
I20260619 01:04:18.808240 18243 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:18.808367 18243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:18.808723 18243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:18.811676 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "f446e7c9c4e949e5be006635d00b4bed"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "86425244d4c5fe2a2f22c8922cc2dbfd"
server_key_iv: "f2ccbc1ebbd48c4b68afd6c3d4850ffd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.812443 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f446e7c9c4e949e5be006635d00b4bed"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "86425244d4c5fe2a2f22c8922cc2dbfd"
server_key_iv: "f2ccbc1ebbd48c4b68afd6c3d4850ffd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.817293 18243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 01:04:18.820684 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.822085 18243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:04:18.822250 18243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "f446e7c9c4e949e5be006635d00b4bed"
format_stamp: "Formatted at 2026-06-19 01:04:18 on dist-test-slave-3rxd"
server_key: "86425244d4c5fe2a2f22c8922cc2dbfd"
server_key_iv: "f2ccbc1ebbd48c4b68afd6c3d4850ffd"
server_key_version: "encryptionkey@0"
I20260619 01:04:18.822470 18243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:18.841970 18243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:18.842756 18243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:18.843030 18243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:18.843928 18243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:18.845373 18243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:18.845499 18243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.845631 18243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:18.845719 18243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:18.877996 18243 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42003
I20260619 01:04:18.878093 18369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42003 every 8 connection(s)
I20260619 01:04:18.879310 18243 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:18.880204 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18243
I20260619 01:04:18.880390 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:18.880973 388 external_mini_cluster.cc:1428] Setting key ac68786efeefd4000508e2b806e8f1d7
I20260619 01:04:18.884357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--builtin_ntp_servers=127.0.97.20:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:18.900696 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:18.882755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59563 (local address 127.0.97.62:40155)
0619 01:04:18.882921 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:18.882927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:18.891767 (+ 8840us) server_negotiation.cc:407] Connection header received
0619 01:04:18.894322 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:18.894331 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:18.894430 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:18.894561 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:18.895663 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.896412 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.897960 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.898358 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.898494 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:18.899893 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:18.899897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:18.899900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:18.899917 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:18.899947 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:18.899958 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:18.900054 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:18.900294 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 01:04:18.900390 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 01:04:18.901753 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:18.883385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40155 (local address 127.0.97.2:59563)
0619 01:04:18.891615 (+ 8230us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:18.891665 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:18.893992 (+ 2327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:18.894831 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:18.894840 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:18.894932 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:18.895493 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:18.895509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.896571 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.896581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:18.897802 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:18.897814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:18.898577 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:18.898584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:18.898866 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:18.899562 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:18.899565 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:18.899602 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:18.899681 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:18.900066 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:18.900078 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:18.900088 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:04:18.900270 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 01:04:18.900643 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7680,"thread_start_us":7492,"threads_started":1}
I20260619 01:04:18.903311 18370 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40155
I20260619 01:04:18.903625 18370 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:18.904307 18370 heartbeater.cc:511] Master 127.0.97.62:40155 requested a full tablet report, sending...
I20260619 01:04:18.905650 18053 ts_manager.cc:194] Registered new tserver with Master: f446e7c9c4e949e5be006635d00b4bed (127.0.97.2:42003)
I20260619 01:04:18.906484 18053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59563
W20260619 01:04:19.211017 18373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:19.211266 18373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:19.211414 18373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:19.218598 18373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:19.218680 18373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:19.218856 18373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:19.227910 18373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:19.229187 18373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:19.230980 18373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:19.243647 18378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:19.243583 18379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:19.243623 18381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:19.244407 18373 server_base.cc:1034] running on GCE node
I20260619 01:04:19.245047 18373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:19.245918 18373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:19.247139 18373 hybrid_clock.cc:648] HybridClock initialized: now 1781831059247102 us; error 53 us; skew 500 ppm
I20260619 01:04:19.247439 18373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:19.252725 18373 webserver.cc:460] Webserver started at http://127.0.97.3:36783/ using document root <none> and password file <none>
I20260619 01:04:19.253369 18373 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:19.253496 18373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:19.253804 18373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:19.256543 18373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "857f9f5879fa4a9d94979c6b738c6013"
format_stamp: "Formatted at 2026-06-19 01:04:19 on dist-test-slave-3rxd"
server_key: "f485a291d877e120e1691114864cec81"
server_key_iv: "47ad4a7384d499f0e3a0b5e9952dc326"
server_key_version: "encryptionkey@0"
I20260619 01:04:19.257294 18373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "857f9f5879fa4a9d94979c6b738c6013"
format_stamp: "Formatted at 2026-06-19 01:04:19 on dist-test-slave-3rxd"
server_key: "f485a291d877e120e1691114864cec81"
server_key_iv: "47ad4a7384d499f0e3a0b5e9952dc326"
server_key_version: "encryptionkey@0"
I20260619 01:04:19.262156 18373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 01:04:19.265406 18387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:19.266628 18373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:04:19.266829 18373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "857f9f5879fa4a9d94979c6b738c6013"
format_stamp: "Formatted at 2026-06-19 01:04:19 on dist-test-slave-3rxd"
server_key: "f485a291d877e120e1691114864cec81"
server_key_iv: "47ad4a7384d499f0e3a0b5e9952dc326"
server_key_version: "encryptionkey@0"
I20260619 01:04:19.267050 18373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:19.283126 18373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:19.283860 18373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:19.284137 18373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:19.285035 18373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:19.286417 18373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:19.286543 18373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:19.286669 18373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:19.286767 18373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:19.321408 18373 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40805
I20260619 01:04:19.321470 18499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40805 every 8 connection(s)
I20260619 01:04:19.323235 18373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:19.324923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18373
I20260619 01:04:19.325094 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:19.325707 388 external_mini_cluster.cc:1428] Setting key deaf88bbf25dcb0acb433b3eac66c6ab
I20260619 01:04:19.340721 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.326878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52571 (local address 127.0.97.62:40155)
0619 01:04:19.327294 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.327300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.329363 (+ 2063us) server_negotiation.cc:407] Connection header received
0619 01:04:19.331832 (+ 2469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.331840 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.331930 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.332035 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:19.333560 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.334922 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.337477 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.338121 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.338260 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.339695 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:19.339703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:19.339711 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:19.339735 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:19.339775 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:19.339816 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:19.339907 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.340223 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.340335 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260619 01:04:19.342342 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.327334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40155 (local address 127.0.97.3:52571)
0619 01:04:19.328815 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.328862 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.331504 (+ 2642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.332338 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.332350 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.332472 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:19.333338 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.333367 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.335112 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.335118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.337307 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.337324 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.338355 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.338360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.338551 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.339292 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:19.339296 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:19.339366 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:19.339472 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:19.339964 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:19.339977 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:19.339988 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.340201 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.340658 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":669,"threads_started":1}
I20260619 01:04:19.344240 18500 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40155
I20260619 01:04:19.344616 18500 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:19.345839 18500 heartbeater.cc:511] Master 127.0.97.62:40155 requested a full tablet report, sending...
I20260619 01:04:19.347605 18053 ts_manager.cc:194] Registered new tserver with Master: 857f9f5879fa4a9d94979c6b738c6013 (127.0.97.3:40805)
I20260619 01:04:19.348913 18053 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:52571
I20260619 01:04:19.359114 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:19.368721 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.363387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.97.62:40155)
0619 01:04:19.363534 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.363539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.363981 (+ 442us) server_negotiation.cc:407] Connection header received
0619 01:04:19.364089 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.364093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.364149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.364225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:19.364664 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.365347 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.366652 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.367117 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.367216 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.368100 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:19.368105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:19.368108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:19.368124 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:19.368154 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:19.368165 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:19.368228 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.368343 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.368487 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 01:04:19.372247 18053 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:19.378859 18053 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:19.410696 18516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.389565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42003 (local address 127.0.0.1:43824)
0619 01:04:19.400342 (+ 10777us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.400361 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.400480 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.402058 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.402069 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.402108 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:19.402515 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.402531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.405072 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.405080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.407870 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.407884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.409693 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.409699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.409920 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.409984 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.410122 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.410302 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10655,"mutex_wait_us":29,"thread_start_us":16423,"threads_started":1}
I20260619 01:04:19.410945 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.389923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.0.97.1:37447)
0619 01:04:19.401784 (+ 11861us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.401790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.401812 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:04:19.401887 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.401892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.402054 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.402280 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:19.404182 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.405195 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.408185 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.409515 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.409636 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.409871 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.410378 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.410591 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11706,"thread_start_us":11602,"threads_started":1}
I20260619 01:04:19.411386 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.389810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.0.97.2:42003)
0619 01:04:19.401012 (+ 11202us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.401024 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.401061 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:04:19.401176 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.401187 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.401499 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.401720 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:19.402780 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.404891 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.408037 (+ 3146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.409502 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.409636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.409877 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.410177 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.410365 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10992,"thread_start_us":11327,"threads_started":1}
I20260619 01:04:19.411839 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.389475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:53984)
0619 01:04:19.390239 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.390277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.390445 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.402986 (+ 12541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.403479 (+ 493us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.403511 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:19.403996 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.404009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.405332 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.405340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.407860 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.407893 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.410011 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.410015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.410180 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.410212 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.411275 (+ 1063us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.411436 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":100,"thread_start_us":899,"threads_started":1}
I20260619 01:04:19.414062 18513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.391211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40805 (local address 127.0.0.1:40940)
0619 01:04:19.406243 (+ 15032us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.406265 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.406393 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.407292 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.407297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.407331 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:19.407735 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.407752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.409153 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.409160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.411124 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.411138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.413447 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.413451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.413562 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.413581 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.413678 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.413790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14903,"thread_start_us":8876,"threads_started":1}
I20260619 01:04:19.414273 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.391944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.0.97.3:40805)
0619 01:04:19.405817 (+ 13873us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.405825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.406376 (+ 551us) server_negotiation.cc:407] Connection header received
0619 01:04:19.406565 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.406573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.406811 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.406985 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:19.407975 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.409018 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.411956 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.413060 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.413138 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.413312 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.413730 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.413893 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13691,"thread_start_us":13521,"threads_started":1}
I20260619 01:04:19.415434 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 (DEFAULT_TABLE table=test-table [id=31a1f13708f3452d812c73f3752876a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:19.415452 18174 tablet_service.cc:1467] Processing CreateTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 (DEFAULT_TABLE table=test-table [id=31a1f13708f3452d812c73f3752876a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:19.416921 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332f0c945ecb45c6b953363a7bb3f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:19.417217 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332f0c945ecb45c6b953363a7bb3f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:19.417555 18435 tablet_service.cc:1467] Processing CreateTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 (DEFAULT_TABLE table=test-table [id=31a1f13708f3452d812c73f3752876a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:19.418635 18435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 332f0c945ecb45c6b953363a7bb3f623. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:19.431528 18519 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Bootstrap starting.
I20260619 01:04:19.434139 18520 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Bootstrap starting.
I20260619 01:04:19.436205 18518 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Bootstrap starting.
I20260619 01:04:19.437105 18519 tablet_bootstrap.cc:654] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:19.437741 18520 tablet_bootstrap.cc:654] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:19.439102 18520 log.cc:824] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:19.439845 18519 log.cc:824] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:19.439901 18518 tablet_bootstrap.cc:654] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:19.441331 18518 log.cc:824] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:19.452296 18518 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: No bootstrap required, opened a new log
I20260619 01:04:19.452739 18518 ts_tablet_manager.cc:1397] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260619 01:04:19.461834 18519 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: No bootstrap required, opened a new log
I20260619 01:04:19.462246 18519 ts_tablet_manager.cc:1397] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.020s
I20260619 01:04:19.462760 18239 heartbeater.cc:503] Master 127.0.97.62:40155 was elected leader, sending a full tablet report...
I20260619 01:04:19.462929 18520 tablet_bootstrap.cc:492] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: No bootstrap required, opened a new log
I20260619 01:04:19.463233 18520 ts_tablet_manager.cc:1397] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.012s
I20260619 01:04:19.465812 18518 raft_consensus.cc:348] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.466295 18518 raft_consensus.cc:374] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:19.466418 18518 raft_consensus.cc:729] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857f9f5879fa4a9d94979c6b738c6013, State: Initialized, Role: FOLLOWER
I20260619 01:04:19.467204 18518 consensus_queue.cc:260] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.471586 18500 heartbeater.cc:503] Master 127.0.97.62:40155 was elected leader, sending a full tablet report...
I20260619 01:04:19.472221 18518 ts_tablet_manager.cc:1428] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Time spent starting tablet: real 0.019s user 0.011s sys 0.006s
I20260619 01:04:19.472438 18520 raft_consensus.cc:348] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.472867 18520 raft_consensus.cc:374] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:19.472997 18520 raft_consensus.cc:729] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f446e7c9c4e949e5be006635d00b4bed, State: Initialized, Role: FOLLOWER
I20260619 01:04:19.473629 18520 consensus_queue.cc:260] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.473702 18519 raft_consensus.cc:348] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.474098 18519 raft_consensus.cc:374] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:19.474270 18519 raft_consensus.cc:729] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5e04e3031764b6d94aaed643ee2605d, State: Initialized, Role: FOLLOWER
I20260619 01:04:19.474815 18370 heartbeater.cc:503] Master 127.0.97.62:40155 was elected leader, sending a full tablet report...
I20260619 01:04:19.475057 18519 consensus_queue.cc:260] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.475467 18520 ts_tablet_manager.cc:1428] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260619 01:04:19.477082 18519 ts_tablet_manager.cc:1428] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260619 01:04:19.576978 18501 tablet.cc:2369] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:19.637874 18524 raft_consensus.cc:482] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260619 01:04:19.638113 18371 tablet.cc:2369] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:19.638238 18524 raft_consensus.cc:504] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.640076 18524 leader_election.cc:288] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5e04e3031764b6d94aaed643ee2605d (127.0.97.1:37447), f446e7c9c4e949e5be006635d00b4bed (127.0.97.2:42003)
I20260619 01:04:19.645597 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.640553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51991 (local address 127.0.97.2:42003)
0619 01:04:19.640730 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.640737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.640758 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:04:19.640887 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.640891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.640965 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.641078 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:19.641556 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.642232 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.643914 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.644739 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.644868 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.644966 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.645217 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.645339 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 01:04:19.645639 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.640443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42003 (local address 127.0.97.3:51991)
0619 01:04:19.640585 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.640606 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.640738 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.641146 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.641151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.641180 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:19.641391 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.641409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.642362 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.642369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.643751 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.643765 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.644897 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.644903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.645065 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.645088 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.645191 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.645322 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260619 01:04:19.646876 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.640516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47375 (local address 127.0.97.1:37447)
0619 01:04:19.640743 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.640750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.641610 (+ 860us) server_negotiation.cc:407] Connection header received
0619 01:04:19.641795 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.641805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.641874 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.641976 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:19.642469 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.643201 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.645023 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.646024 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.646153 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.646239 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.646503 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.646620 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 01:04:19.646675 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332f0c945ecb45c6b953363a7bb3f623" candidate_uuid: "857f9f5879fa4a9d94979c6b738c6013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f446e7c9c4e949e5be006635d00b4bed" is_pre_election: true
I20260619 01:04:19.646950 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.640443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.97.3:47375)
0619 01:04:19.641505 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.641518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:19.641652 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:19.641987 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:19.641991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:19.642012 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:19.642296 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.642304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.643342 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.643347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.644837 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:19.644848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.646185 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.646191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:19.646346 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.646372 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 01:04:19.646477 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:04:19.646591 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"mutex_wait_us":140,"thread_start_us":447,"threads_started":1}
I20260619 01:04:19.647269 18325 raft_consensus.cc:2454] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857f9f5879fa4a9d94979c6b738c6013 in term 0.
I20260619 01:04:19.648010 18391 leader_election.cc:302] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857f9f5879fa4a9d94979c6b738c6013, f446e7c9c4e949e5be006635d00b4bed; no voters:
I20260619 01:04:19.648044 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332f0c945ecb45c6b953363a7bb3f623" candidate_uuid: "857f9f5879fa4a9d94979c6b738c6013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e04e3031764b6d94aaed643ee2605d" is_pre_election: true
I20260619 01:04:19.648438 18524 raft_consensus.cc:2781] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:19.648600 18524 raft_consensus.cc:482] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:19.648557 18194 raft_consensus.cc:2454] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 857f9f5879fa4a9d94979c6b738c6013 in term 0.
I20260619 01:04:19.648722 18524 raft_consensus.cc:3037] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:19.651132 18524 raft_consensus.cc:504] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.651780 18524 leader_election.cc:288] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [CANDIDATE]: Term 1 election: Requested vote from peers d5e04e3031764b6d94aaed643ee2605d (127.0.97.1:37447), f446e7c9c4e949e5be006635d00b4bed (127.0.97.2:42003)
I20260619 01:04:19.652096 18194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332f0c945ecb45c6b953363a7bb3f623" candidate_uuid: "857f9f5879fa4a9d94979c6b738c6013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5e04e3031764b6d94aaed643ee2605d"
I20260619 01:04:19.652292 18194 raft_consensus.cc:3037] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:19.652280 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332f0c945ecb45c6b953363a7bb3f623" candidate_uuid: "857f9f5879fa4a9d94979c6b738c6013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f446e7c9c4e949e5be006635d00b4bed"
I20260619 01:04:19.652494 18325 raft_consensus.cc:3037] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:19.654533 18194 raft_consensus.cc:2454] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857f9f5879fa4a9d94979c6b738c6013 in term 1.
I20260619 01:04:19.654610 18325 raft_consensus.cc:2454] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 857f9f5879fa4a9d94979c6b738c6013 in term 1.
I20260619 01:04:19.654981 18390 leader_election.cc:302] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 857f9f5879fa4a9d94979c6b738c6013, d5e04e3031764b6d94aaed643ee2605d; no voters:
I20260619 01:04:19.655331 18524 raft_consensus.cc:2781] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:19.655771 18524 raft_consensus.cc:686] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 1 LEADER]: Becoming Leader. State: Replica: 857f9f5879fa4a9d94979c6b738c6013, State: Running, Role: LEADER
I20260619 01:04:19.656224 18524 consensus_queue.cc:237] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } }
I20260619 01:04:19.660723 18053 catalog_manager.cc:5495] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 857f9f5879fa4a9d94979c6b738c6013 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "857f9f5879fa4a9d94979c6b738c6013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "857f9f5879fa4a9d94979c6b738c6013" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40805 } health_report { overall_health: HEALTHY } } }
W20260619 01:04:19.692060 18240 tablet.cc:2369] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:19.704023 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.698488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.97.3:40805)
0619 01:04:19.698689 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.698696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.698790 (+ 94us) server_negotiation.cc:407] Connection header received
0619 01:04:19.699055 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.699059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.699143 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.699297 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:19.699651 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.700417 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.701856 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.702267 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.702408 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.702939 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:19.703474 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:19.703639 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.703698 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.703774 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:04:19.711805 18325 raft_consensus.cc:1261] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 1 FOLLOWER]: Refusing update from remote peer 857f9f5879fa4a9d94979c6b738c6013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:19.711862 18194 raft_consensus.cc:1261] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 1 FOLLOWER]: Refusing update from remote peer 857f9f5879fa4a9d94979c6b738c6013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:19.712718 18528 consensus_queue.cc:1035] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f446e7c9c4e949e5be006635d00b4bed" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:19.713135 18524 consensus_queue.cc:1035] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5e04e3031764b6d94aaed643ee2605d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:19.724498 18531 mvcc.cc:205] Tried to move back new op lower bound from 7298380020570075136 to 7298380020355588096. Current Snapshot: MvccSnapshot[applied={T|T < 7298380020570075136}]
I20260619 01:04:19.738423 18053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:19.738731 18053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:19.742527 18053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7331a08ee30a44409fe70020ef9ceaa3: Sending DeleteTablet for 3 replicas of tablet 332f0c945ecb45c6b953363a7bb3f623
I20260619 01:04:19.743796 18305 tablet_service.cc:1514] Processing DeleteTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:19 UTC) from {username='slave'} at 127.0.0.1:43824
I20260619 01:04:19.743891 18174 tablet_service.cc:1514] Processing DeleteTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:19 UTC) from {username='slave'} at 127.0.0.1:53984
I20260619 01:04:19.744211 18435 tablet_service.cc:1514] Processing DeleteTablet for tablet 332f0c945ecb45c6b953363a7bb3f623 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:19 UTC) from {username='slave'} at 127.0.0.1:40940
I20260619 01:04:19.745643 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18112
I20260619 01:04:19.753895 18541 tablet_replica.cc:330] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: stopping tablet replica
I20260619 01:04:19.754417 18541 raft_consensus.cc:2229] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:19.754489 18543 tablet_replica.cc:330] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: stopping tablet replica
I20260619 01:04:19.754909 18543 raft_consensus.cc:2229] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:19.755020 18541 raft_consensus.cc:2258] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:19.755839 18543 raft_consensus.cc:2258] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:19.756719 18542 tablet_replica.cc:330] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: stopping tablet replica
I20260619 01:04:19.757184 18542 raft_consensus.cc:2229] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:19.757741 18542 raft_consensus.cc:2258] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:19.759053 18541 ts_tablet_manager.cc:1905] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:19.759964 18542 ts_tablet_manager.cc:1905] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:19.760684 18543 ts_tablet_manager.cc:1905] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:19.767138 18541 ts_tablet_manager.cc:1918] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:19.767347 18541 log.cc:1196] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/332f0c945ecb45c6b953363a7bb3f623
I20260619 01:04:19.767910 18541 ts_tablet_manager.cc:1939] T 332f0c945ecb45c6b953363a7bb3f623 P f446e7c9c4e949e5be006635d00b4bed: Deleting consensus metadata
I20260619 01:04:19.769627 18040 catalog_manager.cc:4841] TS f446e7c9c4e949e5be006635d00b4bed (127.0.97.2:42003): tablet 332f0c945ecb45c6b953363a7bb3f623 (table test-table [id=31a1f13708f3452d812c73f3752876a9]) successfully deleted
I20260619 01:04:19.772009 18543 ts_tablet_manager.cc:1918] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:19.772265 18543 log.cc:1196] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/332f0c945ecb45c6b953363a7bb3f623
I20260619 01:04:19.772843 18543 ts_tablet_manager.cc:1939] T 332f0c945ecb45c6b953363a7bb3f623 P 857f9f5879fa4a9d94979c6b738c6013: Deleting consensus metadata
I20260619 01:04:19.773047 18515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.746992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.97.1:37447)
0619 01:04:19.747188 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.747194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.761128 (+ 13934us) server_negotiation.cc:407] Connection header received
0619 01:04:19.761348 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.761355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.761456 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.761571 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:19.762064 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.763196 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.765183 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.765830 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.765929 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.771183 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:19.771194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:19.771211 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:19.772054 (+ 843us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:19.772118 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:19.772143 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:19.772246 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.772507 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.772654 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:04:19.773862 18542 ts_tablet_manager.cc:1918] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:19.774107 18542 log.cc:1196] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/332f0c945ecb45c6b953363a7bb3f623
I20260619 01:04:19.774202 18234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:19.775077 18039 catalog_manager.cc:4841] TS 857f9f5879fa4a9d94979c6b738c6013 (127.0.97.3:40805): tablet 332f0c945ecb45c6b953363a7bb3f623 (table test-table [id=31a1f13708f3452d812c73f3752876a9]) successfully deleted
I20260619 01:04:19.888520 18542 ts_tablet_manager.cc:1939] T 332f0c945ecb45c6b953363a7bb3f623 P d5e04e3031764b6d94aaed643ee2605d: Deleting consensus metadata
I20260619 01:04:19.888902 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18112
W20260619 01:04:19.912228 18039 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:37447 (error 108)
I20260619 01:04:19.913462 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18243
I20260619 01:04:19.913841 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.913349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54000)
0619 01:04:19.913552 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.913677 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 01:04:19.914080 18039 catalog_manager.cc:4584] TS d5e04e3031764b6d94aaed643ee2605d (127.0.97.1:37447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 332f0c945ecb45c6b953363a7bb3f623: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
I20260619 01:04:19.920993 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.914360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.0.97.2:42003)
0619 01:04:19.914565 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:19.914571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:19.914591 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:04:19.914654 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:19.914659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:19.914819 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:19.914927 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:19.915289 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.916444 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.917665 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:19.918292 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:19.918415 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:19.919489 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:19.919500 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:19.919517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:19.920267 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:19.920326 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:19.920349 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:19.920471 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:19.920555 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 01:04:19.920654 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:04:19.921592 18365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:19.967862 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.967297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54010)
0619 01:04:19.967551 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.967667 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 01:04:19.969081 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:19.968643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54020)
0619 01:04:19.968831 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:19.968920 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:04:20.026306 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18243
I20260619 01:04:20.029613 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.029039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54030)
0619 01:04:20.029272 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:20.029398 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 01:04:20.030936 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.030405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54044)
0619 01:04:20.030660 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:20.030788 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 01:04:20.043949 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18373
I20260619 01:04:20.051025 18517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.044844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40950 (local address 127.0.97.3:40805)
0619 01:04:20.045024 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:20.045030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:20.045077 (+ 47us) server_negotiation.cc:407] Connection header received
0619 01:04:20.045295 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:20.045300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:20.045374 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:20.045507 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:20.045890 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:20.046690 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:20.048034 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:20.048477 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:20.048612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:20.049658 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:20.049666 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:20.049679 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:20.050249 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:20.050292 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:20.050309 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:20.050438 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:20.050596 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 01:04:20.050689 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 01:04:20.051898 18495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:20.111892 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.111323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54046)
0619 01:04:20.111563 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:20.111687 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 01:04:20.112965 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.112556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54048)
0619 01:04:20.112751 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:20.112839 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 01:04:20.169276 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18373
I20260619 01:04:20.188199 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18022
I20260619 01:04:20.189271 18083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:20.295415 18512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:20.294473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37447 (local address 127.0.0.1:54054)
0619 01:04:20.295134 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:20.295237 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491}
I20260619 01:04:20.295413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18022
2026-06-19T01:04:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:20 dist-test-slave-3rxd krb5kdc[18552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:20 dist-test-slave-3rxd krb5kdc[18552](info): set up 2 sockets
Jun 19 01:04:20 dist-test-slave-3rxd krb5kdc[18552](info): commencing operation
krb5kdc: starting...
W20260619 01:04:21.374548 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:21 dist-test-slave-3rxd krb5kdc[18552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:04:21.599138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43807
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:21.913941 18565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:21.914207 18565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:21.914379 18565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:21.921289 18565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:21.921413 18565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:21.921496 18565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:21.921594 18565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:21.921682 18565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:21.931349 18565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:21.932627 18565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:21.933940 18565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:21.945413 18570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:21.945480 18571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:21.945456 18573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:21.946352 18565 server_base.cc:1034] running on GCE node
I20260619 01:04:21.947173 18565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:21.948495 18565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:21.949694 18565 hybrid_clock.cc:648] HybridClock initialized: now 1781831061949661 us; error 35 us; skew 500 ppm
I20260619 01:04:21.950044 18565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:21 dist-test-slave-3rxd krb5kdc[18552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:21.953043 18565 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:04:21.955034 18565 webserver.cc:460] Webserver started at http://127.0.97.62:46381/ using document root <none> and password file <none>
I20260619 01:04:21.955721 18565 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:21.955842 18565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:21.956153 18565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:21.959038 18565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "95da1a3315ff43238ada6cb7a3abcfed"
format_stamp: "Formatted at 2026-06-19 01:04:21 on dist-test-slave-3rxd"
server_key: "6808ff462a0af9bce970badd4f7df847"
server_key_iv: "1a2222d6900d6507a213fa6f0c5fcb33"
server_key_version: "encryptionkey@0"
I20260619 01:04:21.959774 18565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "95da1a3315ff43238ada6cb7a3abcfed"
format_stamp: "Formatted at 2026-06-19 01:04:21 on dist-test-slave-3rxd"
server_key: "6808ff462a0af9bce970badd4f7df847"
server_key_iv: "1a2222d6900d6507a213fa6f0c5fcb33"
server_key_version: "encryptionkey@0"
I20260619 01:04:21.964593 18565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:21.967832 18580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:21.969118 18565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:04:21.969297 18565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "95da1a3315ff43238ada6cb7a3abcfed"
format_stamp: "Formatted at 2026-06-19 01:04:21 on dist-test-slave-3rxd"
server_key: "6808ff462a0af9bce970badd4f7df847"
server_key_iv: "1a2222d6900d6507a213fa6f0c5fcb33"
server_key_version: "encryptionkey@0"
I20260619 01:04:21.969484 18565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:21.987244 18565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:21.990932 18565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:21.991201 18565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:22.010610 18565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43807
I20260619 01:04:22.010636 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43807 every 8 connection(s)
I20260619 01:04:22.011965 18565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:22.015694 18632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:22.019130 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18565
I20260619 01:04:22.019310 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:22.019827 388 external_mini_cluster.cc:1428] Setting key 4222d56c0020d396c35a90f76557d26d
I20260619 01:04:22.028024 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: Bootstrap starting.
I20260619 01:04:22.031651 18632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:22.032941 18632 log.cc:824] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:22.035822 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: No bootstrap required, opened a new log
Jun 19 01:04:22 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:22.047472 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:22.021817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33848 (local address 127.0.97.62:43807)
0619 01:04:22.022688 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:22.022714 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:22.023162 (+ 448us) server_negotiation.cc:407] Connection header received
0619 01:04:22.026296 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:22.026326 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:22.026841 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:22.027293 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:22.028662 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.030665 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.032052 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.033132 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.033308 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:22.037418 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:22.037430 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:22.037454 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:22.037500 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:22.041634 (+ 4134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:22.042648 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:22.042655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:22.042678 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:22.042996 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:22.043462 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:22.043471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:22.043475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:22.044270 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:22.044582 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:22.045359 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 01:04:22.045985 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":351,"threads_started":1}
I20260619 01:04:22.047564 18632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260619 01:04:22.048049 18632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:22.048174 18632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95da1a3315ff43238ada6cb7a3abcfed, State: Initialized, Role: FOLLOWER
I20260619 01:04:22.048908 18632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260619 01:04:22.049242 18632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:22.049408 18632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:22.049685 18632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:22.052337 18632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260619 01:04:22.052850 18632 leader_election.cc:302] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95da1a3315ff43238ada6cb7a3abcfed; no voters:
I20260619 01:04:22.053369 18632 leader_election.cc:288] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:22.053508 18637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:22.054190 18637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [term 1 LEADER]: Becoming Leader. State: Replica: 95da1a3315ff43238ada6cb7a3abcfed, State: Running, Role: LEADER
I20260619 01:04:22.054764 18637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } }
I20260619 01:04:22.055516 18632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:22.058305 18638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95da1a3315ff43238ada6cb7a3abcfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } } }
I20260619 01:04:22.058763 18638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:22.058265 18639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95da1a3315ff43238ada6cb7a3abcfed. Latest consensus state: current_term: 1 leader_uuid: "95da1a3315ff43238ada6cb7a3abcfed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95da1a3315ff43238ada6cb7a3abcfed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43807 } } }
I20260619 01:04:22.059993 18639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:22.066161 18642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:22.070798 18642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:22.079476 18642 catalog_manager.cc:1347] Generated new cluster ID: 895bf8b87531478eb5d7dea6af8bf128
I20260619 01:04:22.079592 18642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:22.095464 18642 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:22.096706 18642 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:22.111038 18642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: Generated new TSK 0
I20260619 01:04:22.111835 18642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:04:22.169840 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:22.485756 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:22.486073 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:22.486254 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:22.493800 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:22.493954 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:22.494154 18658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:22.503821 18658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:22.505044 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:22.506305 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:22.519429 18663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:22.519380 18664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:22.519404 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:22.520268 18658 server_base.cc:1034] running on GCE node
I20260619 01:04:22.520875 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:22.521822 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:22.523061 18658 hybrid_clock.cc:648] HybridClock initialized: now 1781831062523020 us; error 28 us; skew 500 ppm
I20260619 01:04:22.523432 18658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:22 dist-test-slave-3rxd krb5kdc[18552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:22.526600 18658 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:04:22.528069 18658 webserver.cc:460] Webserver started at http://127.0.97.1:43307/ using document root <none> and password file <none>
I20260619 01:04:22.528770 18658 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:22.529006 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:22.529361 18658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:22.532398 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "c9ec87e775a84f54aa6bc657defd50ae"
format_stamp: "Formatted at 2026-06-19 01:04:22 on dist-test-slave-3rxd"
server_key: "70a5c0863bedc1dca1e399f472539461"
server_key_iv: "1a101339317896ffd4d6f8e3097c2ddd"
server_key_version: "encryptionkey@0"
I20260619 01:04:22.533185 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c9ec87e775a84f54aa6bc657defd50ae"
format_stamp: "Formatted at 2026-06-19 01:04:22 on dist-test-slave-3rxd"
server_key: "70a5c0863bedc1dca1e399f472539461"
server_key_iv: "1a101339317896ffd4d6f8e3097c2ddd"
server_key_version: "encryptionkey@0"
I20260619 01:04:22.538439 18658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:04:22.541980 18673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:22.543351 18658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:04:22.543536 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "c9ec87e775a84f54aa6bc657defd50ae"
format_stamp: "Formatted at 2026-06-19 01:04:22 on dist-test-slave-3rxd"
server_key: "70a5c0863bedc1dca1e399f472539461"
server_key_iv: "1a101339317896ffd4d6f8e3097c2ddd"
server_key_version: "encryptionkey@0"
I20260619 01:04:22.543756 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:22.557228 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:22.561029 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:22.561302 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:22.562170 18658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:22.563477 18658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:22.563580 18658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:22.563726 18658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:22.563802 18658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:22.595191 18658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45773
I20260619 01:04:22.595285 18785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45773 every 8 connection(s)
I20260619 01:04:22.596451 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:22.604823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18658
I20260619 01:04:22.605007 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:22.605572 388 external_mini_cluster.cc:1428] Setting key 5a8feaac11c7ebf68bc9b3de5879be4b
Jun 19 01:04:22 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:22.616387 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:22.598924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43199 (local address 127.0.97.62:43807)
0619 01:04:22.599346 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:22.599352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:22.600314 (+ 962us) server_negotiation.cc:407] Connection header received
0619 01:04:22.602023 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:22.602027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:22.602090 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:22.602183 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:22.603577 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.604275 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.606408 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.606930 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.607024 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:22.610867 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:22.610872 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:22.610875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:22.610916 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:22.613334 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:22.614054 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:22.614064 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:22.614066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:22.614176 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:22.614621 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:22.614626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:22.614628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:22.614982 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:22.615141 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:22.615783 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 01:04:22.616015 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":192,"threads_started":1}
I20260619 01:04:22.617266 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:22.599505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.1:43199)
0619 01:04:22.600159 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:22.600196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:22.601776 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:22.602364 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:22.602371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:22.602914 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:22.603439 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:22.603455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.604388 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.604391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:22.606277 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:22.606289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:22.607134 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:22.607141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:22.607356 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:22.608007 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:22.608034 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:22.610608 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:22.613488 (+ 2880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:22.613496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:22.613511 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:22.613927 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:22.614290 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:22.614294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:22.614298 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:22.614505 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:22.615112 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:22.615119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:22.615444 (+ 325us) client_negotiation.cc:769] Sending connection context
0619 01:04:22.615691 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 01:04:22.616187 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":271,"threads_started":1}
I20260619 01:04:22.618777 18786 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260619 01:04:22.619091 18786 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:22.619925 18786 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
I20260619 01:04:22.622174 18597 ts_manager.cc:194] Registered new tserver with Master: c9ec87e775a84f54aa6bc657defd50ae (127.0.97.1:45773)
I20260619 01:04:22.624249 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43199
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:04:22.661163 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:22.973147 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:22.973433 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:22.973604 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:22.980947 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:22.981057 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:22.981222 18792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:22.990329 18792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:22.991546 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:22.992761 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:23.005074 18797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:23.005368 18800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:23.005096 18798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:23.006330 18792 server_base.cc:1034] running on GCE node
I20260619 01:04:23.007050 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:23.008173 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:23.009382 18792 hybrid_clock.cc:648] HybridClock initialized: now 1781831063009323 us; error 48 us; skew 500 ppm
I20260619 01:04:23.009634 18792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:23 dist-test-slave-3rxd krb5kdc[18552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:23.012917 18792 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:04:23.014305 18792 webserver.cc:460] Webserver started at http://127.0.97.2:45841/ using document root <none> and password file <none>
I20260619 01:04:23.015017 18792 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:23.015185 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:23.015507 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:23.018172 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "420c5222bdce4f5194cde3659c65eced"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "8f7fe2ba9d7ac2502041c8ec90505705"
server_key_iv: "e517d113ca78f15df5e050db748c2759"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.018980 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "420c5222bdce4f5194cde3659c65eced"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "8f7fe2ba9d7ac2502041c8ec90505705"
server_key_iv: "e517d113ca78f15df5e050db748c2759"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.023631 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 01:04:23.026782 18807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.027983 18792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 01:04:23.028168 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "420c5222bdce4f5194cde3659c65eced"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "8f7fe2ba9d7ac2502041c8ec90505705"
server_key_iv: "e517d113ca78f15df5e050db748c2759"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.028355 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:23.049089 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:23.053332 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:23.053601 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:23.054462 18792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:23.055806 18792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:23.055910 18792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.056072 18792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:23.056144 18792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.087807 18792 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37055
I20260619 01:04:23.087900 18919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37055 every 8 connection(s)
I20260619 01:04:23.089020 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:23.093994 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18792
I20260619 01:04:23.094165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:23.094578 388 external_mini_cluster.cc:1428] Setting key a555c890b750e87a0a6be2c6ba7a7d2f
Jun 19 01:04:23 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:23.110090 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.091558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58197 (local address 127.0.97.62:43807)
0619 01:04:23.091732 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.091737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.092835 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 01:04:23.094570 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.094577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.094664 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.094797 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:23.096294 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.097145 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.099656 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.100024 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.100111 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.104047 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:23.104056 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:23.104063 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:23.104109 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:23.107085 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.107974 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.107978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.107980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.108102 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.108558 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.108562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.108564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.108801 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:23.108917 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.109709 (+ 792us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.109868 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:04:23.111706 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.092009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.2:58197)
0619 01:04:23.092682 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.092717 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.094326 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.094990 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.094998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.095577 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:23.096128 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.096144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.097312 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.097320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.099504 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.099522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.100229 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.100238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.100553 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.101241 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:23.101274 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:23.103824 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:23.107208 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:23.107219 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:23.107236 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:23.107863 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:23.108205 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:23.108212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:23.108218 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:23.108457 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:23.108937 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:23.108948 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:23.109346 (+ 398us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.109636 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.110166 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":264,"threads_started":1}
I20260619 01:04:23.113166 18920 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260619 01:04:23.113452 18920 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:23.114106 18920 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
I20260619 01:04:23.115538 18597 ts_manager.cc:194] Registered new tserver with Master: 420c5222bdce4f5194cde3659c65eced (127.0.97.2:37055)
I20260619 01:04:23.116309 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58197
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:04:23.149358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:23.459167 18925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:23.459420 18925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:23.459510 18925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:23.466393 18925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:23.466488 18925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:23.466603 18925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:23.475852 18925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:23.477044 18925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:23.478298 18925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:23.490756 18930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:23.490783 18931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:23.490686 18933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:23.491487 18925 server_base.cc:1034] running on GCE node
I20260619 01:04:23.492107 18925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:23.492986 18925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:23.494204 18925 hybrid_clock.cc:648] HybridClock initialized: now 1781831063494156 us; error 62 us; skew 500 ppm
I20260619 01:04:23.494506 18925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:23 dist-test-slave-3rxd krb5kdc[18552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:23.497581 18925 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:04:23.499084 18925 webserver.cc:460] Webserver started at http://127.0.97.3:34243/ using document root <none> and password file <none>
I20260619 01:04:23.499761 18925 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:23.499898 18925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:23.500196 18925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:23.503070 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "110d1bd6d01d4e759becb592f219b6ec"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "54d34e92710644abee348f961fe89761"
server_key_iv: "f39da8e3e75229b45ce4e2a864430890"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.503870 18925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "110d1bd6d01d4e759becb592f219b6ec"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "54d34e92710644abee348f961fe89761"
server_key_iv: "f39da8e3e75229b45ce4e2a864430890"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.508813 18925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 01:04:23.512619 18940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.514307 18925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:04:23.514480 18925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "110d1bd6d01d4e759becb592f219b6ec"
format_stamp: "Formatted at 2026-06-19 01:04:23 on dist-test-slave-3rxd"
server_key: "54d34e92710644abee348f961fe89761"
server_key_iv: "f39da8e3e75229b45ce4e2a864430890"
server_key_version: "encryptionkey@0"
I20260619 01:04:23.514704 18925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:23.546118 18925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:23.550474 18925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:23.550820 18925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:23.551789 18925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:23.553074 18925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:23.553174 18925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.553349 18925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:23.553409 18925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:23.585326 18925 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42131
I20260619 01:04:23.585417 19052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42131 every 8 connection(s)
I20260619 01:04:23.586596 18925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:23.590652 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 18925
I20260619 01:04:23.590857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:23.591320 388 external_mini_cluster.cc:1428] Setting key 7ef964b85b2c6e81c41ea5bc35c2bd4b
Jun 19 01:04:23 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:23.608001 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.589192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34161 (local address 127.0.97.62:43807)
0619 01:04:23.589388 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.589398 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.590570 (+ 1172us) server_negotiation.cc:407] Connection header received
0619 01:04:23.593012 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.593018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.593113 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.593219 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:23.595128 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.596319 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.597838 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.598476 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.598605 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.601995 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:23.602000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:23.602003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:23.602036 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:23.604689 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.605726 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.605731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.605735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.605894 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.606353 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.606357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.606359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.606620 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:23.606803 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.607441 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.607663 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:04:23.608892 19055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.589700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43807 (local address 127.0.97.3:34161)
0619 01:04:23.590372 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.590430 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.592706 (+ 2276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.593501 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.593516 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.594249 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:23.594958 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.594978 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.596449 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.596453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.597684 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.597696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.598687 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.598692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.598951 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.599616 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:23.599637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:23.601804 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:23.605049 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:23.605057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:23.605070 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:23.605626 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:23.605998 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:23.606002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:23.606005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:23.606212 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:23.606759 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:23.606766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:23.607071 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.607331 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.607818 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":265,"threads_started":1}
I20260619 01:04:23.610113 19053 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43807
I20260619 01:04:23.610407 19053 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:23.611150 19053 heartbeater.cc:511] Master 127.0.97.62:43807 requested a full tablet report, sending...
I20260619 01:04:23.612478 18597 ts_manager.cc:194] Registered new tserver with Master: 110d1bd6d01d4e759becb592f219b6ec (127.0.97.3:42131)
I20260619 01:04:23.613270 18597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34161
I20260619 01:04:23.618259 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:23.627853 18786 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
I20260619 01:04:23.634775 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.622086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.0.97.62:43807)
0619 01:04:23.622230 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.622234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.622689 (+ 455us) server_negotiation.cc:407] Connection header received
0619 01:04:23.622858 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.622861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.622910 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.622981 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:23.624251 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.624956 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.626239 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.626669 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.626852 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.629638 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:23.629643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:23.629646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:23.629673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:23.632288 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.633122 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.633125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.633127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.633234 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.633628 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.633631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.633633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.633896 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:23.634076 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.634387 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.634508 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 01:04:23.638321 18597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:23.644775 18597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:23.672211 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.656430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42131 (local address 127.0.0.1:54884)
0619 01:04:23.657474 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.657525 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.657688 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.664024 (+ 6336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.664033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.664063 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:23.664514 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.664525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.666871 (+ 2346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.666878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.669363 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.669379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.671211 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.671217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.671436 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.671506 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.671608 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.671781 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":30,"thread_start_us":6737,"threads_started":1}
I20260619 01:04:23.673307 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.656673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.0.97.3:42131)
0619 01:04:23.663392 (+ 6719us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.663401 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.663435 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:04:23.663541 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.663551 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.663784 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.664729 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:23.664999 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.666680 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.669541 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.670969 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.672462 (+ 1493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.672678 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.672826 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.672985 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6568,"thread_start_us":6387,"threads_started":1}
I20260619 01:04:23.674453 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.657436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.0.97.1:45773)
0619 01:04:23.663030 (+ 5594us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.663040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.663532 (+ 492us) server_negotiation.cc:407] Connection header received
0619 01:04:23.663778 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.663785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.664009 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.664158 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:23.666174 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.667720 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.671437 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.672993 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.673092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.673302 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.673950 (+ 648us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.674123 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5419,"thread_start_us":5282,"threads_started":1}
I20260619 01:04:23.675258 19065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.656733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45773 (local address 127.0.0.1:44774)
0619 01:04:23.663423 (+ 6690us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.663445 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.663621 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.664476 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.664480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.664508 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:23.664938 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.664951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.667915 (+ 2964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.667920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.671252 (+ 3332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.671268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.673576 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.673585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.673758 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.673788 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.674730 (+ 942us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.674862 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6568,"mutex_wait_us":84,"thread_start_us":1030,"threads_started":1}
I20260619 01:04:23.677440 18988 tablet_service.cc:1467] Processing CreateTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 (DEFAULT_TABLE table=test-table [id=167b1ef819a44c6fa16946aa2afb0a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:23.678941 18988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2631ba28f7248bf82bb1e8f9e443597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:23.678483 18721 tablet_service.cc:1467] Processing CreateTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 (DEFAULT_TABLE table=test-table [id=167b1ef819a44c6fa16946aa2afb0a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:23.679919 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2631ba28f7248bf82bb1e8f9e443597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:23.683533 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.663543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37055 (local address 127.0.0.1:40154)
0619 01:04:23.671564 (+ 8021us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.671593 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.671743 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.676527 (+ 4784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.676531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.676559 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:23.676843 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.676854 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.678653 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.678657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.680722 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.680737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.682779 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.682786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.682957 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.682989 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.683084 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.683211 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7837,"thread_start_us":8294,"threads_started":1}
I20260619 01:04:23.683590 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.663790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.0.97.2:37055)
0619 01:04:23.675985 (+ 12195us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.675993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.676020 (+ 27us) server_negotiation.cc:407] Connection header received
0619 01:04:23.676115 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.676121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.676330 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.676535 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:23.677010 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.678490 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.681164 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.682559 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.682687 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.682873 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.683142 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.683294 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12045,"thread_start_us":11945,"threads_started":1}
I20260619 01:04:23.689679 18855 tablet_service.cc:1467] Processing CreateTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 (DEFAULT_TABLE table=test-table [id=167b1ef819a44c6fa16946aa2afb0a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:23.691056 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2631ba28f7248bf82bb1e8f9e443597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:23.691238 19071 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Bootstrap starting.
I20260619 01:04:23.692878 19072 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Bootstrap starting.
I20260619 01:04:23.694702 19071 tablet_bootstrap.cc:654] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:23.695742 19071 log.cc:824] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:23.696125 19072 tablet_bootstrap.cc:654] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:23.697254 19072 log.cc:824] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:23.699949 19072 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: No bootstrap required, opened a new log
I20260619 01:04:23.700037 19071 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: No bootstrap required, opened a new log
I20260619 01:04:23.700218 19072 ts_tablet_manager.cc:1397] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 01:04:23.700273 19071 ts_tablet_manager.cc:1397] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 01:04:23.704712 19075 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Bootstrap starting.
I20260619 01:04:23.707921 19075 tablet_bootstrap.cc:654] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:23.708549 19072 raft_consensus.cc:348] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.708948 19072 raft_consensus.cc:374] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:23.709044 19072 raft_consensus.cc:729] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ec87e775a84f54aa6bc657defd50ae, State: Initialized, Role: FOLLOWER
I20260619 01:04:23.709105 19075 log.cc:824] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:23.709143 19071 raft_consensus.cc:348] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.709587 19071 raft_consensus.cc:374] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:23.709702 19071 raft_consensus.cc:729] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110d1bd6d01d4e759becb592f219b6ec, State: Initialized, Role: FOLLOWER
I20260619 01:04:23.709697 19072 consensus_queue.cc:260] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.710270 19071 consensus_queue.cc:260] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.711757 19072 ts_tablet_manager.cc:1428] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260619 01:04:23.712910 19053 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
I20260619 01:04:23.713521 19071 ts_tablet_manager.cc:1428] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260619 01:04:23.714537 19075 tablet_bootstrap.cc:492] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: No bootstrap required, opened a new log
I20260619 01:04:23.714866 19075 ts_tablet_manager.cc:1397] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260619 01:04:23.722262 19075 raft_consensus.cc:348] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.722618 19075 raft_consensus.cc:374] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:23.722749 19075 raft_consensus.cc:729] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420c5222bdce4f5194cde3659c65eced, State: Initialized, Role: FOLLOWER
I20260619 01:04:23.723229 19075 consensus_queue.cc:260] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.724159 18920 heartbeater.cc:503] Master 127.0.97.62:43807 was elected leader, sending a full tablet report...
I20260619 01:04:23.724715 19075 ts_tablet_manager.cc:1428] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260619 01:04:23.773317 19079 raft_consensus.cc:482] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:04:23.773643 19079 raft_consensus.cc:504] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.775416 19079 leader_election.cc:288] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110d1bd6d01d4e759becb592f219b6ec (127.0.97.3:42131), c9ec87e775a84f54aa6bc657defd50ae (127.0.97.1:45773)
I20260619 01:04:23.782694 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.775742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34259 (local address 127.0.97.3:42131)
0619 01:04:23.775959 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.775968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.776827 (+ 859us) server_negotiation.cc:407] Connection header received
0619 01:04:23.776984 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.776990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.777075 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.777163 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:23.777818 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.778657 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.780861 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.781691 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.781805 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.781945 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.782308 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.782431 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:04:23.782699 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.775822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33707 (local address 127.0.97.1:45773)
0619 01:04:23.776038 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.776046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.776998 (+ 952us) server_negotiation.cc:407] Connection header received
0619 01:04:23.777179 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.777184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.777262 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.777368 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:23.777996 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.778698 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.780904 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.781708 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.781920 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.782018 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.782338 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.782459 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:04:23.782895 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.775767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42131 (local address 127.0.97.2:34259)
0619 01:04:23.776712 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.776727 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.776842 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.777198 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.777202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.777224 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:23.777623 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.777633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.778885 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.778890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.780708 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.780719 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.781958 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.781965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.782135 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.782176 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.782246 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.782405 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"mutex_wait_us":6,"thread_start_us":507,"threads_started":1}
I20260619 01:04:23.782920 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.775843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45773 (local address 127.0.97.2:33707)
0619 01:04:23.776863 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:23.776881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:23.777024 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:23.777389 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:23.777392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:23.777411 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:23.777833 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.777844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.778845 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.778853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.780714 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:23.780725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.782005 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.782010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:23.782138 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.782186 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 01:04:23.782289 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 01:04:23.782407 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":119,"thread_start_us":458,"threads_started":1}
I20260619 01:04:23.783835 19008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2631ba28f7248bf82bb1e8f9e443597" candidate_uuid: "420c5222bdce4f5194cde3659c65eced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110d1bd6d01d4e759becb592f219b6ec" is_pre_election: true
I20260619 01:04:23.784201 18741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2631ba28f7248bf82bb1e8f9e443597" candidate_uuid: "420c5222bdce4f5194cde3659c65eced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ec87e775a84f54aa6bc657defd50ae" is_pre_election: true
I20260619 01:04:23.784467 19008 raft_consensus.cc:2454] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420c5222bdce4f5194cde3659c65eced in term 0.
I20260619 01:04:23.784713 18741 raft_consensus.cc:2454] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 420c5222bdce4f5194cde3659c65eced in term 0.
I20260619 01:04:23.785245 18810 leader_election.cc:302] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110d1bd6d01d4e759becb592f219b6ec, 420c5222bdce4f5194cde3659c65eced; no voters:
I20260619 01:04:23.785616 19079 raft_consensus.cc:2781] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:23.785755 19079 raft_consensus.cc:482] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:23.785858 19079 raft_consensus.cc:3037] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:23.788087 19079 raft_consensus.cc:504] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.788726 19079 leader_election.cc:288] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [CANDIDATE]: Term 1 election: Requested vote from peers 110d1bd6d01d4e759becb592f219b6ec (127.0.97.3:42131), c9ec87e775a84f54aa6bc657defd50ae (127.0.97.1:45773)
I20260619 01:04:23.789103 19008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2631ba28f7248bf82bb1e8f9e443597" candidate_uuid: "420c5222bdce4f5194cde3659c65eced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110d1bd6d01d4e759becb592f219b6ec"
I20260619 01:04:23.789177 18741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2631ba28f7248bf82bb1e8f9e443597" candidate_uuid: "420c5222bdce4f5194cde3659c65eced" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9ec87e775a84f54aa6bc657defd50ae"
I20260619 01:04:23.789327 19008 raft_consensus.cc:3037] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:23.789400 18741 raft_consensus.cc:3037] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:23.791602 19008 raft_consensus.cc:2454] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420c5222bdce4f5194cde3659c65eced in term 1.
I20260619 01:04:23.791633 18741 raft_consensus.cc:2454] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 420c5222bdce4f5194cde3659c65eced in term 1.
I20260619 01:04:23.791990 18810 leader_election.cc:302] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110d1bd6d01d4e759becb592f219b6ec, 420c5222bdce4f5194cde3659c65eced; no voters:
I20260619 01:04:23.792245 19079 raft_consensus.cc:2781] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:23.792649 19079 raft_consensus.cc:686] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 1 LEADER]: Becoming Leader. State: Replica: 420c5222bdce4f5194cde3659c65eced, State: Running, Role: LEADER
I20260619 01:04:23.793093 19079 consensus_queue.cc:237] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } }
I20260619 01:04:23.797284 18597 catalog_manager.cc:5495] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced reported cstate change: term changed from 0 to 1, leader changed from <none> to 420c5222bdce4f5194cde3659c65eced (127.0.97.2). New cstate: current_term: 1 leader_uuid: "420c5222bdce4f5194cde3659c65eced" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "420c5222bdce4f5194cde3659c65eced" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37055 } health_report { overall_health: HEALTHY } } }
I20260619 01:04:23.819981 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.815025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.0.97.2:37055)
0619 01:04:23.815190 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.815195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.815311 (+ 116us) server_negotiation.cc:407] Connection header received
0619 01:04:23.815451 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.815454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.815524 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.815640 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:23.815950 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.816631 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.817875 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.818244 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.818327 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.818815 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:23.819383 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:23.819515 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.819632 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.819687 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:04:23.827507 19008 raft_consensus.cc:1261] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 1 FOLLOWER]: Refusing update from remote peer 420c5222bdce4f5194cde3659c65eced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:23.827502 18741 raft_consensus.cc:1261] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 1 FOLLOWER]: Refusing update from remote peer 420c5222bdce4f5194cde3659c65eced: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:23.828608 19082 consensus_queue.cc:1035] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [LEADER]: Connected to new peer: Peer: permanent_uuid: "110d1bd6d01d4e759becb592f219b6ec" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:23.828944 19079 consensus_queue.cc:1035] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ec87e775a84f54aa6bc657defd50ae" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 01:04:23.838399 19054 tablet.cc:2369] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:23.851862 18597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:23.852121 18597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:23.855574 18597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 95da1a3315ff43238ada6cb7a3abcfed: Sending DeleteTablet for 3 replicas of tablet f2631ba28f7248bf82bb1e8f9e443597
I20260619 01:04:23.856681 18988 tablet_service.cc:1514] Processing DeleteTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54884
I20260619 01:04:23.856752 18855 tablet_service.cc:1514] Processing DeleteTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40154
I20260619 01:04:23.856729 18721 tablet_service.cc:1514] Processing DeleteTablet for tablet f2631ba28f7248bf82bb1e8f9e443597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44774
I20260619 01:04:23.863278 19095 tablet_replica.cc:330] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: stopping tablet replica
I20260619 01:04:23.863334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18658
I20260619 01:04:23.863394 19094 tablet_replica.cc:330] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: stopping tablet replica
I20260619 01:04:23.863759 19094 raft_consensus.cc:2229] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:23.863780 19095 raft_consensus.cc:2229] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:23.864262 19094 raft_consensus.cc:2258] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:23.864544 19095 raft_consensus.cc:2258] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:23.866300 19096 tablet_replica.cc:330] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: stopping tablet replica
I20260619 01:04:23.866434 19094 ts_tablet_manager.cc:1905] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:23.866498 19095 ts_tablet_manager.cc:1905] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:23.866671 19096 raft_consensus.cc:2229] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:23.873872 19095 ts_tablet_manager.cc:1918] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:23.874042 19095 log.cc:1196] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/f2631ba28f7248bf82bb1e8f9e443597
I20260619 01:04:23.874615 19095 ts_tablet_manager.cc:1939] T f2631ba28f7248bf82bb1e8f9e443597 P 420c5222bdce4f5194cde3659c65eced: Deleting consensus metadata
I20260619 01:04:23.875075 19096 raft_consensus.cc:2258] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:23.875324 19094 ts_tablet_manager.cc:1918] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:23.875499 19094 log.cc:1196] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/f2631ba28f7248bf82bb1e8f9e443597
I20260619 01:04:23.876053 19094 ts_tablet_manager.cc:1939] T f2631ba28f7248bf82bb1e8f9e443597 P 110d1bd6d01d4e759becb592f219b6ec: Deleting consensus metadata
I20260619 01:04:23.876376 18583 catalog_manager.cc:4841] TS 420c5222bdce4f5194cde3659c65eced (127.0.97.2:37055): tablet f2631ba28f7248bf82bb1e8f9e443597 (table test-table [id=167b1ef819a44c6fa16946aa2afb0a31]) successfully deleted
I20260619 01:04:23.876997 19096 ts_tablet_manager.cc:1905] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:23.877660 18583 catalog_manager.cc:4841] TS 110d1bd6d01d4e759becb592f219b6ec (127.0.97.3:42131): tablet f2631ba28f7248bf82bb1e8f9e443597 (table test-table [id=167b1ef819a44c6fa16946aa2afb0a31]) successfully deleted
I20260619 01:04:23.883930 19096 ts_tablet_manager.cc:1918] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:23.884104 19096 log.cc:1196] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/f2631ba28f7248bf82bb1e8f9e443597
I20260619 01:04:23.884593 19096 ts_tablet_manager.cc:1939] T f2631ba28f7248bf82bb1e8f9e443597 P c9ec87e775a84f54aa6bc657defd50ae: Deleting consensus metadata
Jun 19 01:04:23 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:04:23.886410 18582 catalog_manager.cc:4841] TS c9ec87e775a84f54aa6bc657defd50ae (127.0.97.1:45773): tablet f2631ba28f7248bf82bb1e8f9e443597 (table test-table [id=167b1ef819a44c6fa16946aa2afb0a31]) successfully deleted
I20260619 01:04:23.891220 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.865808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.97.1:45773)
0619 01:04:23.865996 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.866003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.876833 (+ 10830us) server_negotiation.cc:407] Connection header received
0619 01:04:23.877115 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.877120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.877201 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.877271 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:23.878918 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.879694 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.881027 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:23.881457 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:23.881592 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:23.885608 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:23.885621 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:23.885637 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:23.885676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:23.888538 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.889294 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.889301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.889303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.889429 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:23.889809 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:23.889812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:23.889813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:23.890114 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:23.890317 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:23.890758 (+ 441us) server_negotiation.cc:299] Negotiation successful
0619 01:04:23.890926 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:04:23.891904 18781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:23.980394 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18658
I20260619 01:04:23.997852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18792
Jun 19 01:04:24 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:04:24.016331 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:23.998637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.0.97.2:37055)
0619 01:04:23.998823 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:23.998831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:23.998851 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:04:23.999070 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:23.999082 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:23.999175 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:23.999288 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:24.000572 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.001607 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.003111 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.003749 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.003875 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:24.007943 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:24.007961 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:24.007983 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:24.008032 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:24.012955 (+ 4923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:24.013773 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:24.013783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:24.013787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:24.013961 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:24.014368 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:24.014373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:24.014377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:24.014853 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:24.015202 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:24.015491 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 01:04:24.015759 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:04:24.017128 18915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:24.130084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18792
I20260619 01:04:24.149034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18925
Jun 19 01:04:24 dist-test-slave-3rxd krb5kdc[18552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:04:24.163470 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:24.149768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.0.97.3:42131)
0619 01:04:24.149950 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:24.149954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:24.149971 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:04:24.150113 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:24.150117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:24.150190 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:24.150273 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:24.151605 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.152337 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.153572 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.153956 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.154047 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:24.158464 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:24.158478 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:24.158495 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:24.158528 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:24.161097 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:24.161722 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:24.161728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:24.161730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:24.161860 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:24.162188 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:24.162191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:24.162193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:24.162522 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:24.162901 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:24.163062 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 01:04:24.163229 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:04:24.164127 19048 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:24.251780 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18925
I20260619 01:04:24.268424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 18565
I20260619 01:04:24.269448 18627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:24.378522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 18565
2026-06-19T01:04:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4084 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T01:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:24Z Disabled control of system clock
I20260619 01:04:24.452673 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34725
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:24.765746 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:24.766040 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:24.766202 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:24.773391 19105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:24.773491 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:24.773559 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:24.773640 19105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:24.773715 19105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:24.782698 19105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:24.783982 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:24.785742 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:24.796949 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:24.796964 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:24.796959 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:24.797848 19105 server_base.cc:1034] running on GCE node
I20260619 01:04:24.798583 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:24.799839 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:24.801087 19105 hybrid_clock.cc:648] HybridClock initialized: now 1781831064801040 us; error 43 us; skew 500 ppm
I20260619 01:04:24.801422 19105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:24.808003 19105 webserver.cc:460] Webserver started at http://127.0.97.62:43407/ using document root <none> and password file <none>
I20260619 01:04:24.808656 19105 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:24.808787 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:24.809106 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:24.811889 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "240ab6b88c0343798a9caf1d0b719acc"
format_stamp: "Formatted at 2026-06-19 01:04:24 on dist-test-slave-3rxd"
server_key: "b55909f7b6885257e9d94b9182115426"
server_key_iv: "e31443bda3ad67f8f4661e5582aca306"
server_key_version: "encryptionkey@0"
I20260619 01:04:24.812616 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "240ab6b88c0343798a9caf1d0b719acc"
format_stamp: "Formatted at 2026-06-19 01:04:24 on dist-test-slave-3rxd"
server_key: "b55909f7b6885257e9d94b9182115426"
server_key_iv: "e31443bda3ad67f8f4661e5582aca306"
server_key_version: "encryptionkey@0"
I20260619 01:04:24.817346 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:24.820595 19119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:24.821906 19105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:04:24.822079 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "240ab6b88c0343798a9caf1d0b719acc"
format_stamp: "Formatted at 2026-06-19 01:04:24 on dist-test-slave-3rxd"
server_key: "b55909f7b6885257e9d94b9182115426"
server_key_iv: "e31443bda3ad67f8f4661e5582aca306"
server_key_version: "encryptionkey@0"
I20260619 01:04:24.822280 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:24.836522 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:24.837222 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:24.837497 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:24.855156 19105 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34725
I20260619 01:04:24.855190 19170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34725 every 8 connection(s)
I20260619 01:04:24.856453 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:24.859867 19171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:24.861758 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19105
I20260619 01:04:24.861933 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:24.862450 388 external_mini_cluster.cc:1428] Setting key 9f7323dd9ca2787dc3f361bba83b7e0c
I20260619 01:04:24.871066 19171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: Bootstrap starting.
I20260619 01:04:24.874125 19171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:24.875233 19171 log.cc:824] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:24.875550 19174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:24.864326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.0.97.62:34725)
0619 01:04:24.865268 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:24.865282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:24.865319 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:04:24.867050 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:24.867071 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:24.867517 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:24.867869 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:24.868434 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.869730 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.871135 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:24.871832 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:24.872042 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:24.873186 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:24.873193 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:24.873205 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:24.873701 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:24.873787 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:24.873812 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:24.873936 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:24.874187 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 01:04:24.874515 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260619 01:04:24.878046 19171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: No bootstrap required, opened a new log
I20260619 01:04:24.884868 19171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } }
I20260619 01:04:24.885277 19171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:24.885406 19171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240ab6b88c0343798a9caf1d0b719acc, State: Initialized, Role: FOLLOWER
I20260619 01:04:24.885977 19171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } }
I20260619 01:04:24.886214 19171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:24.886356 19171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:24.886533 19171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:24.888583 19171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } }
I20260619 01:04:24.889035 19171 leader_election.cc:302] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240ab6b88c0343798a9caf1d0b719acc; no voters:
I20260619 01:04:24.889586 19171 leader_election.cc:288] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:24.889731 19176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:24.890401 19176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [term 1 LEADER]: Becoming Leader. State: Replica: 240ab6b88c0343798a9caf1d0b719acc, State: Running, Role: LEADER
I20260619 01:04:24.890954 19176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } }
I20260619 01:04:24.891585 19171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:24.893392 19177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240ab6b88c0343798a9caf1d0b719acc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } } }
I20260619 01:04:24.893790 19177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:24.893797 19178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240ab6b88c0343798a9caf1d0b719acc. Latest consensus state: current_term: 1 leader_uuid: "240ab6b88c0343798a9caf1d0b719acc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ab6b88c0343798a9caf1d0b719acc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34725 } } }
I20260619 01:04:24.894488 19178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:24.895821 19181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:24.901201 19181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:24.908974 19181 catalog_manager.cc:1347] Generated new cluster ID: 20134bc7bb9d490c9a6e0a5f27ea4790
I20260619 01:04:24.909072 19181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:24.935678 19181 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:24.936892 19181 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:24.944659 19181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: Generated new TSK 0
I20260619 01:04:24.945652 19181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:24.957798 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--builtin_ntp_servers=127.0.97.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:25.305225 19195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:25.305491 19195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:25.305652 19195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:25.312472 19195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:25.312585 19195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:25.312763 19195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:25.321403 19195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:25.322537 19195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:25.324268 19195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:25.336974 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:25.336939 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:25.337020 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:25.337703 19195 server_base.cc:1034] running on GCE node
I20260619 01:04:25.338253 19195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:25.339190 19195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:25.340382 19195 hybrid_clock.cc:648] HybridClock initialized: now 1781831065340353 us; error 27 us; skew 500 ppm
I20260619 01:04:25.340673 19195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:25.342643 19195 webserver.cc:460] Webserver started at http://127.0.97.1:42111/ using document root <none> and password file <none>
I20260619 01:04:25.343286 19195 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:25.343416 19195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:25.343719 19195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:25.346392 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "1142c5f94ebb4c6b952a786c1b19c43a"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "8b122edc7389839283daf2db964d98db"
server_key_iv: "82d25a6043c861b1f39dc460694ba27e"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.347170 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1142c5f94ebb4c6b952a786c1b19c43a"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "8b122edc7389839283daf2db964d98db"
server_key_iv: "82d25a6043c861b1f39dc460694ba27e"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.351847 19195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 01:04:25.355086 19209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.356274 19195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:04:25.356550 19195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "1142c5f94ebb4c6b952a786c1b19c43a"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "8b122edc7389839283daf2db964d98db"
server_key_iv: "82d25a6043c861b1f39dc460694ba27e"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.356761 19195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:25.385723 19195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:25.386581 19195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:25.386888 19195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:25.387815 19195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:25.389245 19195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:25.389377 19195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.389503 19195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:25.389588 19195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.421123 19195 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40917
I20260619 01:04:25.421207 19321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40917 every 8 connection(s)
I20260619 01:04:25.422369 19195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:25.423722 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19195
I20260619 01:04:25.423897 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:25.424450 388 external_mini_cluster.cc:1428] Setting key a13804f659a3a9b8a9f0d8f1bc67b2f1
I20260619 01:04:25.427779 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--builtin_ntp_servers=127.0.97.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:25.435406 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:25.425132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37297 (local address 127.0.97.62:34725)
0619 01:04:25.427162 (+ 2030us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:25.427167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:25.427186 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:04:25.428954 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:25.428962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:25.429050 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:25.429187 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:25.430259 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.431102 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.432638 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.433041 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.433178 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:25.434518 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:25.434525 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:25.434531 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:25.434553 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:25.434592 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:25.434604 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:25.434699 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:25.434946 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 01:04:25.435039 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":2813,"threads_started":1}
I20260619 01:04:25.436592 19325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:25.425779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34725 (local address 127.0.97.1:37297)
0619 01:04:25.426985 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:25.427038 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:25.428690 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:25.429437 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:25.429446 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:25.429533 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:25.430089 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:25.430106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.431263 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.431269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:25.432470 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:25.432481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.433256 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.433265 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:25.433499 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:25.434188 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:25.434191 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:25.434227 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:25.434297 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:25.434700 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:25.434733 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:25.434743 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:04:25.434928 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 01:04:25.435304 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":354,"threads_started":1}
I20260619 01:04:25.438133 19322 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34725
I20260619 01:04:25.438418 19322 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:25.439154 19322 heartbeater.cc:511] Master 127.0.97.62:34725 requested a full tablet report, sending...
I20260619 01:04:25.441108 19136 ts_manager.cc:194] Registered new tserver with Master: 1142c5f94ebb4c6b952a786c1b19c43a (127.0.97.1:40917)
I20260619 01:04:25.442571 19136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:37297
W20260619 01:04:25.749284 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:25.749585 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:25.749748 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:25.756834 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:25.756978 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:25.757153 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:25.766317 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:25.767557 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:25.769315 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:25.782114 19331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:25.782052 19334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:25.782564 19332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:25.782843 19326 server_base.cc:1034] running on GCE node
I20260619 01:04:25.783407 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:25.784300 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:25.785579 19326 hybrid_clock.cc:648] HybridClock initialized: now 1781831065785508 us; error 48 us; skew 500 ppm
I20260619 01:04:25.785912 19326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:25.788163 19326 webserver.cc:460] Webserver started at http://127.0.97.2:39545/ using document root <none> and password file <none>
I20260619 01:04:25.788837 19326 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:25.788973 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:25.789283 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:25.792189 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "35f741c52d174c0b900611f2e98d2b05"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "90a08149093f0a7e2feab18209e4d218"
server_key_iv: "ad5f23c2295e994e856b38ef3b3c4f01"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.792948 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "35f741c52d174c0b900611f2e98d2b05"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "90a08149093f0a7e2feab18209e4d218"
server_key_iv: "ad5f23c2295e994e856b38ef3b3c4f01"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.797801 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260619 01:04:25.801019 19340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.802175 19326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:04:25.802345 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "35f741c52d174c0b900611f2e98d2b05"
format_stamp: "Formatted at 2026-06-19 01:04:25 on dist-test-slave-3rxd"
server_key: "90a08149093f0a7e2feab18209e4d218"
server_key_iv: "ad5f23c2295e994e856b38ef3b3c4f01"
server_key_version: "encryptionkey@0"
I20260619 01:04:25.802558 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:25.819844 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:25.820602 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:25.820873 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:25.821774 19326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:25.823269 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:25.823397 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.823524 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:25.823607 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:25.856982 19326 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37433
I20260619 01:04:25.857019 19452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37433 every 8 connection(s)
I20260619 01:04:25.858249 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:25.865917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19326
I20260619 01:04:25.866118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:25.866578 388 external_mini_cluster.cc:1428] Setting key ba8aab632315205405c09ba823cef832
I20260619 01:04:25.869699 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--builtin_ntp_servers=127.0.97.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:25.872779 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:25.861162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55267 (local address 127.0.97.62:34725)
0619 01:04:25.861385 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:25.861395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:25.863006 (+ 1611us) server_negotiation.cc:407] Connection header received
0619 01:04:25.865515 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:25.865522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:25.865618 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:25.865747 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:25.867201 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.868184 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.869971 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.870410 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.870483 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:25.871991 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:25.871997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:25.872001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:25.872028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:25.872057 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:25.872068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:25.872161 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:25.872381 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 01:04:25.872474 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:04:25.873836 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:25.861742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34725 (local address 127.0.97.2:55267)
0619 01:04:25.862833 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:25.862883 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:25.865175 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:25.866032 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:25.866045 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:25.866170 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:25.866964 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:25.866991 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.868424 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.868431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:25.869764 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:25.869802 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:25.870784 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:25.870790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:25.870991 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:25.871691 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:25.871694 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:25.871731 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:25.871802 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:25.872161 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:25.872174 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:25.872184 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 01:04:25.872357 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 01:04:25.872741 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":300,"threads_started":1}
I20260619 01:04:25.875224 19453 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34725
I20260619 01:04:25.875535 19453 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:25.876255 19453 heartbeater.cc:511] Master 127.0.97.62:34725 requested a full tablet report, sending...
I20260619 01:04:25.877597 19136 ts_manager.cc:194] Registered new tserver with Master: 35f741c52d174c0b900611f2e98d2b05 (127.0.97.2:37433)
I20260619 01:04:25.878386 19136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:55267
W20260619 01:04:26.188159 19456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:26.188476 19456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:26.188675 19456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:26.196012 19456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:26.196138 19456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:26.196285 19456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:26.206001 19456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:26.207414 19456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:26.209193 19456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:26.221949 19462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:26.222057 19461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:26.222093 19464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:26.222784 19456 server_base.cc:1034] running on GCE node
I20260619 01:04:26.223388 19456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:26.224284 19456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:26.225566 19456 hybrid_clock.cc:648] HybridClock initialized: now 1781831066225509 us; error 54 us; skew 500 ppm
I20260619 01:04:26.225898 19456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:26.228089 19456 webserver.cc:460] Webserver started at http://127.0.97.3:39757/ using document root <none> and password file <none>
I20260619 01:04:26.228747 19456 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:26.228892 19456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:26.229202 19456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:26.232034 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "10231c90ad584f3db3508de23e0189fe"
format_stamp: "Formatted at 2026-06-19 01:04:26 on dist-test-slave-3rxd"
server_key: "4acffb746b7dd17c41ccff0ee27a9ef6"
server_key_iv: "ae3064f16be869dfce1c3956663bfc34"
server_key_version: "encryptionkey@0"
I20260619 01:04:26.232761 19456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "10231c90ad584f3db3508de23e0189fe"
format_stamp: "Formatted at 2026-06-19 01:04:26 on dist-test-slave-3rxd"
server_key: "4acffb746b7dd17c41ccff0ee27a9ef6"
server_key_iv: "ae3064f16be869dfce1c3956663bfc34"
server_key_version: "encryptionkey@0"
I20260619 01:04:26.237454 19456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:26.241205 19470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:26.242604 19456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:04:26.242790 19456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "10231c90ad584f3db3508de23e0189fe"
format_stamp: "Formatted at 2026-06-19 01:04:26 on dist-test-slave-3rxd"
server_key: "4acffb746b7dd17c41ccff0ee27a9ef6"
server_key_iv: "ae3064f16be869dfce1c3956663bfc34"
server_key_version: "encryptionkey@0"
I20260619 01:04:26.243022 19456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:26.256733 19456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:26.257498 19456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:26.257779 19456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:26.258688 19456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:26.260195 19456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:26.260309 19456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:26.260439 19456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:26.260533 19456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:26.292253 19456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40395
I20260619 01:04:26.292348 19582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40395 every 8 connection(s)
I20260619 01:04:26.293622 19456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:26.294281 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19456
I20260619 01:04:26.294459 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:26.295079 388 external_mini_cluster.cc:1428] Setting key 60e5d15e4157fb566be6d524c850b4dc
I20260619 01:04:26.305691 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.296433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53027 (local address 127.0.97.62:34725)
0619 01:04:26.296620 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.296626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.298310 (+ 1684us) server_negotiation.cc:407] Connection header received
0619 01:04:26.300042 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.300046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.300109 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.300204 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:26.301183 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.301899 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.303420 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.303801 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.303889 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.304982 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:26.304987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:26.304990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:26.305005 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:26.305035 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:26.305045 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:26.305110 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.305286 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.305355 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:04:26.306898 19585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.297009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34725 (local address 127.0.97.3:53027)
0619 01:04:26.298142 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.298178 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.299815 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.300393 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.300400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.300478 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:26.301036 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.301052 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.302026 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.302030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.303300 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.303311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.303977 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.303981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.304156 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.304726 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:26.304728 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:26.304761 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:26.304824 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:26.305124 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:26.305130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:26.305135 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.305258 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.305578 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":413,"threads_started":1}
I20260619 01:04:26.308199 19583 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34725
I20260619 01:04:26.308488 19583 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:26.309388 19583 heartbeater.cc:511] Master 127.0.97.62:34725 requested a full tablet report, sending...
I20260619 01:04:26.310855 19136 ts_manager.cc:194] Registered new tserver with Master: 10231c90ad584f3db3508de23e0189fe (127.0.97.3:40395)
I20260619 01:04:26.311578 19136 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:53027
I20260619 01:04:26.322166 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:26.331506 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.326082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.0.97.62:34725)
0619 01:04:26.326297 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.326301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.326759 (+ 458us) server_negotiation.cc:407] Connection header received
0619 01:04:26.326915 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.326919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.326987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.327068 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:26.327454 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.328151 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.329449 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.329821 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.329903 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.330937 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:26.330941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:26.330944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:26.330960 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:26.330989 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:26.330999 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:26.331102 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.331252 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.331292 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 01:04:26.335059 19136 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:26.341892 19136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:26.368330 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.353684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.0.97.1:40917)
0619 01:04:26.355187 (+ 1503us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.355199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.356378 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 01:04:26.356771 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.356781 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.357043 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.357230 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:26.358506 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.360578 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.364354 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.366336 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.366483 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.366820 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.367766 (+ 946us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.367967 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1285,"thread_start_us":1151,"threads_started":1}
I20260619 01:04:26.368484 19597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.353473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40917 (local address 127.0.0.1:42420)
0619 01:04:26.356227 (+ 2754us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.356265 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.356492 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.357630 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.357642 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.357684 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:26.358251 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.358268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.360851 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.360858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.364058 (+ 3200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.364087 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.367047 (+ 2960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.367056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.367441 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.367528 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.367706 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.367956 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2502,"mutex_wait_us":1,"thread_start_us":2812,"threads_started":1}
I20260619 01:04:26.371929 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.354987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34660 (local address 127.0.97.2:37433)
0619 01:04:26.356109 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.356121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.358078 (+ 1957us) server_negotiation.cc:407] Connection header received
0619 01:04:26.358465 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.358476 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.358793 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.358976 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:26.360167 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.363767 (+ 3600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.368977 (+ 5210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.370373 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.370525 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.370865 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.371382 (+ 517us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.371551 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":761,"threads_started":1}
I20260619 01:04:26.372749 19595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.354259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37433 (local address 127.0.0.1:34660)
0619 01:04:26.356599 (+ 2340us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.356623 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.358226 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.359366 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.359373 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.359422 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:26.359913 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.359930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.363977 (+ 4047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.363989 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.368752 (+ 4763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.368767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.371003 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.371009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.371180 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.371213 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.372180 (+ 967us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.372407 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2174,"mutex_wait_us":754,"thread_start_us":1238,"threads_started":1}
I20260619 01:04:26.372774 19257 tablet_service.cc:1467] Processing CreateTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 (DEFAULT_TABLE table=test-table [id=bd428d6f2677466dbbc3a55f224a73be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:26.374226 19257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d67c8489b449a182ccb539d9d2a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:26.376005 19599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.356675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40395 (local address 127.0.0.1:57940)
0619 01:04:26.367770 (+ 11095us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.367788 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.367893 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.369585 (+ 1692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.369590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.369615 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:26.369961 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.369972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.371852 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.371858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.373478 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.373494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.375271 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.375277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.375473 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.375504 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.375606 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.375731 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10967,"thread_start_us":10824,"threads_started":1}
I20260619 01:04:26.376142 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.356975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.0.97.3:40395)
0619 01:04:26.369084 (+ 12109us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.369091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.369126 (+ 35us) server_negotiation.cc:407] Connection header received
0619 01:04:26.369219 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.369224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.369399 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.369605 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:26.370142 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.371714 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.373668 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.375082 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.375236 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.375514 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.375641 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.375787 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11926,"thread_start_us":11970,"threads_started":1}
I20260619 01:04:26.376416 19388 tablet_service.cc:1467] Processing CreateTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 (DEFAULT_TABLE table=test-table [id=bd428d6f2677466dbbc3a55f224a73be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:26.377779 19388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d67c8489b449a182ccb539d9d2a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:26.379650 19518 tablet_service.cc:1467] Processing CreateTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 (DEFAULT_TABLE table=test-table [id=bd428d6f2677466dbbc3a55f224a73be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:26.380939 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d67c8489b449a182ccb539d9d2a5e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:26.385702 19601 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Bootstrap starting.
I20260619 01:04:26.389285 19601 tablet_bootstrap.cc:654] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:26.390250 19601 log.cc:824] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:26.392985 19603 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Bootstrap starting.
I20260619 01:04:26.393213 19601 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: No bootstrap required, opened a new log
I20260619 01:04:26.393491 19601 ts_tablet_manager.cc:1397] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 01:04:26.394610 19604 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Bootstrap starting.
I20260619 01:04:26.397598 19604 tablet_bootstrap.cc:654] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:26.397693 19603 tablet_bootstrap.cc:654] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:26.398523 19604 log.cc:824] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:26.398968 19603 log.cc:824] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:26.401536 19604 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: No bootstrap required, opened a new log
I20260619 01:04:26.401793 19604 ts_tablet_manager.cc:1397] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20260619 01:04:26.402998 19603 tablet_bootstrap.cc:492] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: No bootstrap required, opened a new log
I20260619 01:04:26.403267 19603 ts_tablet_manager.cc:1397] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260619 01:04:26.404160 19601 raft_consensus.cc:348] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.404592 19601 raft_consensus.cc:374] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:26.404691 19601 raft_consensus.cc:729] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1142c5f94ebb4c6b952a786c1b19c43a, State: Initialized, Role: FOLLOWER
I20260619 01:04:26.405282 19601 consensus_queue.cc:260] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.407667 19322 heartbeater.cc:503] Master 127.0.97.62:34725 was elected leader, sending a full tablet report...
I20260619 01:04:26.408313 19601 ts_tablet_manager.cc:1428] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260619 01:04:26.412184 19603 raft_consensus.cc:348] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.412551 19603 raft_consensus.cc:374] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:26.412668 19603 raft_consensus.cc:729] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35f741c52d174c0b900611f2e98d2b05, State: Initialized, Role: FOLLOWER
I20260619 01:04:26.412482 19604 raft_consensus.cc:348] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.412915 19604 raft_consensus.cc:374] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:26.413012 19604 raft_consensus.cc:729] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10231c90ad584f3db3508de23e0189fe, State: Initialized, Role: FOLLOWER
I20260619 01:04:26.413291 19603 consensus_queue.cc:260] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.413622 19604 consensus_queue.cc:260] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.414485 19453 heartbeater.cc:503] Master 127.0.97.62:34725 was elected leader, sending a full tablet report...
I20260619 01:04:26.414916 19583 heartbeater.cc:503] Master 127.0.97.62:34725 was elected leader, sending a full tablet report...
I20260619 01:04:26.415181 19603 ts_tablet_manager.cc:1428] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260619 01:04:26.415512 19604 ts_tablet_manager.cc:1428] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260619 01:04:26.424551 19323 tablet.cc:2369] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:26.445535 19608 raft_consensus.cc:482] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:04:26.445806 19608 raft_consensus.cc:504] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.447672 19608 leader_election.cc:288] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1142c5f94ebb4c6b952a786c1b19c43a (127.0.97.1:40917), 10231c90ad584f3db3508de23e0189fe (127.0.97.3:40395)
I20260619 01:04:26.453879 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.447983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41853 (local address 127.0.97.1:40917)
0619 01:04:26.448153 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.448158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.448731 (+ 573us) server_negotiation.cc:407] Connection header received
0619 01:04:26.448992 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.449000 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.449102 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.449199 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:26.449733 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.450437 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.452294 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.453019 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.453151 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.453259 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.453520 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.453637 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:04:26.454092 19610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.447982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40917 (local address 127.0.97.2:41853)
0619 01:04:26.448626 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.448647 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.448835 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.449252 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.449256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.449279 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:26.449587 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.449595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.450586 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.450592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.452110 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.452121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.453210 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.453216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.453365 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.453388 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.453494 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.453613 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":3,"thread_start_us":338,"threads_started":1}
I20260619 01:04:26.454491 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.448848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52863 (local address 127.0.97.3:40395)
0619 01:04:26.449097 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.449103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.449480 (+ 377us) server_negotiation.cc:407] Connection header received
0619 01:04:26.449682 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.449686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.449738 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.449828 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:26.450327 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.451097 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.452808 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.453536 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.453665 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.453773 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.454128 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.454268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 01:04:26.455092 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.448659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40395 (local address 127.0.97.2:52863)
0619 01:04:26.449375 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:26.449388 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:26.449537 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:26.449841 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:26.449845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:26.449866 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:26.450185 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.450192 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.451223 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.451230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.452631 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:26.452643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.453724 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.453730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:26.453900 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.453928 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 01:04:26.454652 (+ 724us) client_negotiation.cc:240] Negotiation successful
0619 01:04:26.454808 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":36,"thread_start_us":385,"threads_started":1}
I20260619 01:04:26.455080 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d67c8489b449a182ccb539d9d2a5e4" candidate_uuid: "35f741c52d174c0b900611f2e98d2b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" is_pre_election: true
I20260619 01:04:26.455642 19277 raft_consensus.cc:2454] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f741c52d174c0b900611f2e98d2b05 in term 0.
I20260619 01:04:26.456244 19344 leader_election.cc:302] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1142c5f94ebb4c6b952a786c1b19c43a, 35f741c52d174c0b900611f2e98d2b05; no voters:
I20260619 01:04:26.456162 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d67c8489b449a182ccb539d9d2a5e4" candidate_uuid: "35f741c52d174c0b900611f2e98d2b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10231c90ad584f3db3508de23e0189fe" is_pre_election: true
I20260619 01:04:26.456609 19608 raft_consensus.cc:2781] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:26.456614 19538 raft_consensus.cc:2454] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f741c52d174c0b900611f2e98d2b05 in term 0.
I20260619 01:04:26.456717 19608 raft_consensus.cc:482] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:26.456857 19608 raft_consensus.cc:3037] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:26.459162 19608 raft_consensus.cc:504] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.459694 19608 leader_election.cc:288] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [CANDIDATE]: Term 1 election: Requested vote from peers 1142c5f94ebb4c6b952a786c1b19c43a (127.0.97.1:40917), 10231c90ad584f3db3508de23e0189fe (127.0.97.3:40395)
I20260619 01:04:26.460155 19277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d67c8489b449a182ccb539d9d2a5e4" candidate_uuid: "35f741c52d174c0b900611f2e98d2b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1142c5f94ebb4c6b952a786c1b19c43a"
I20260619 01:04:26.460268 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d67c8489b449a182ccb539d9d2a5e4" candidate_uuid: "35f741c52d174c0b900611f2e98d2b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10231c90ad584f3db3508de23e0189fe"
I20260619 01:04:26.460361 19277 raft_consensus.cc:3037] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:26.460505 19538 raft_consensus.cc:3037] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:26.462636 19277 raft_consensus.cc:2454] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f741c52d174c0b900611f2e98d2b05 in term 1.
I20260619 01:04:26.462690 19538 raft_consensus.cc:2454] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f741c52d174c0b900611f2e98d2b05 in term 1.
I20260619 01:04:26.462998 19344 leader_election.cc:302] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1142c5f94ebb4c6b952a786c1b19c43a, 35f741c52d174c0b900611f2e98d2b05; no voters:
I20260619 01:04:26.463256 19608 raft_consensus.cc:2781] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:26.463627 19608 raft_consensus.cc:686] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 1 LEADER]: Becoming Leader. State: Replica: 35f741c52d174c0b900611f2e98d2b05, State: Running, Role: LEADER
I20260619 01:04:26.464075 19608 consensus_queue.cc:237] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } }
I20260619 01:04:26.468375 19136 catalog_manager.cc:5495] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35f741c52d174c0b900611f2e98d2b05 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "35f741c52d174c0b900611f2e98d2b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f741c52d174c0b900611f2e98d2b05" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37433 } health_report { overall_health: HEALTHY } } }
I20260619 01:04:26.490371 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.485484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.0.97.2:37433)
0619 01:04:26.485703 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.485708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.485726 (+ 18us) server_negotiation.cc:407] Connection header received
0619 01:04:26.485875 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.485879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.485954 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.486074 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:26.486390 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.487107 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.488410 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.488814 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.488902 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.489326 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:26.489792 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:26.489948 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.490022 (+ 74us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.490083 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:04:26.498698 19277 raft_consensus.cc:1261] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 1 FOLLOWER]: Refusing update from remote peer 35f741c52d174c0b900611f2e98d2b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:26.498698 19538 raft_consensus.cc:1261] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 1 FOLLOWER]: Refusing update from remote peer 35f741c52d174c0b900611f2e98d2b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:26.499722 19612 consensus_queue.cc:1035] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1142c5f94ebb4c6b952a786c1b19c43a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:26.500087 19608 consensus_queue.cc:1035] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10231c90ad584f3db3508de23e0189fe" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:26.525548 19135 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:26.525847 19135 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:26.529510 19135 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 240ab6b88c0343798a9caf1d0b719acc: Sending DeleteTablet for 3 replicas of tablet 34d67c8489b449a182ccb539d9d2a5e4
I20260619 01:04:26.530606 19257 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:26 UTC) from {username='slave'} at 127.0.0.1:42420
I20260619 01:04:26.530743 19518 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:26 UTC) from {username='slave'} at 127.0.0.1:57940
I20260619 01:04:26.531597 19624 tablet_replica.cc:330] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: stopping tablet replica
I20260619 01:04:26.531808 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19195
I20260619 01:04:26.531786 19388 tablet_service.cc:1514] Processing DeleteTablet for tablet 34d67c8489b449a182ccb539d9d2a5e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:26 UTC) from {username='slave'} at 127.0.0.1:34660
I20260619 01:04:26.532204 19624 raft_consensus.cc:2229] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:26.532753 19624 raft_consensus.cc:2258] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:26.539563 19626 tablet_replica.cc:330] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: stopping tablet replica
I20260619 01:04:26.540074 19626 raft_consensus.cc:2229] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:26.540817 19626 raft_consensus.cc:2258] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:26.542759 19624 ts_tablet_manager.cc:1905] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:26.542850 19626 ts_tablet_manager.cc:1905] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:26.550447 19626 ts_tablet_manager.cc:1918] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:26.550756 19626 log.cc:1196] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/34d67c8489b449a182ccb539d9d2a5e4
I20260619 01:04:26.551417 19626 ts_tablet_manager.cc:1939] T 34d67c8489b449a182ccb539d9d2a5e4 P 35f741c52d174c0b900611f2e98d2b05: Deleting consensus metadata
I20260619 01:04:26.553352 19122 catalog_manager.cc:4841] TS 35f741c52d174c0b900611f2e98d2b05 (127.0.97.2:37433): tablet 34d67c8489b449a182ccb539d9d2a5e4 (table test-table [id=bd428d6f2677466dbbc3a55f224a73be]) successfully deleted
I20260619 01:04:26.554464 19625 tablet_replica.cc:330] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: stopping tablet replica
I20260619 01:04:26.555022 19625 raft_consensus.cc:2229] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:26.555604 19625 raft_consensus.cc:2258] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:26.557163 19624 ts_tablet_manager.cc:1918] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:26.557331 19624 log.cc:1196] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/34d67c8489b449a182ccb539d9d2a5e4
I20260619 01:04:26.557711 19625 ts_tablet_manager.cc:1905] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:26.557878 19624 ts_tablet_manager.cc:1939] T 34d67c8489b449a182ccb539d9d2a5e4 P 1142c5f94ebb4c6b952a786c1b19c43a: Deleting consensus metadata
I20260619 01:04:26.559744 19123 catalog_manager.cc:4841] TS 1142c5f94ebb4c6b952a786c1b19c43a (127.0.97.1:40917): tablet 34d67c8489b449a182ccb539d9d2a5e4 (table test-table [id=bd428d6f2677466dbbc3a55f224a73be]) successfully deleted
I20260619 01:04:26.562409 19596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.533548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.0.97.1:40917)
0619 01:04:26.533734 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.533740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.548485 (+ 14745us) server_negotiation.cc:407] Connection header received
0619 01:04:26.553980 (+ 5495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.553989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.554125 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.554238 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:26.554804 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.555915 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.557963 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.558639 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.558774 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.560557 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:26.560568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:26.560585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:26.561356 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:26.561439 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:26.561473 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:26.561590 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.561768 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.561891 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:04:26.563103 19317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:26.564908 19625 ts_tablet_manager.cc:1918] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:26.565100 19625 log.cc:1196] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/34d67c8489b449a182ccb539d9d2a5e4
I20260619 01:04:26.565667 19625 ts_tablet_manager.cc:1939] T 34d67c8489b449a182ccb539d9d2a5e4 P 10231c90ad584f3db3508de23e0189fe: Deleting consensus metadata
I20260619 01:04:26.567283 19123 catalog_manager.cc:4841] TS 10231c90ad584f3db3508de23e0189fe (127.0.97.3:40395): tablet 34d67c8489b449a182ccb539d9d2a5e4 (table test-table [id=bd428d6f2677466dbbc3a55f224a73be]) successfully deleted
I20260619 01:04:26.684120 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19195
I20260619 01:04:26.702878 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19326
I20260619 01:04:26.710997 19598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.703684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.0.97.2:37433)
0619 01:04:26.703913 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.703919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.703941 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:04:26.704084 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.704090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.704214 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.704325 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:26.704671 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.705861 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.707462 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.708129 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.708265 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.709287 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:26.709304 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:26.709324 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:26.710156 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:26.710224 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:26.710248 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:26.710354 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.710463 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.710570 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 01:04:26.711803 19448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:26.826803 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19326
I20260619 01:04:26.845712 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19456
I20260619 01:04:26.852170 19600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:26.846503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57942 (local address 127.0.97.3:40395)
0619 01:04:26.846700 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:26.846738 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:26.846837 (+ 99us) server_negotiation.cc:407] Connection header received
0619 01:04:26.846988 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:26.846991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:26.847064 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:26.847142 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:26.847473 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.848227 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.849388 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:26.849737 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:26.849820 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:26.851004 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:26.851013 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:26.851027 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:26.851604 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:26.851653 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:26.851672 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:26.851755 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:26.851863 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 01:04:26.851939 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:04:26.852772 19578 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:26.953930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19456
I20260619 01:04:26.970780 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19105
I20260619 01:04:26.971861 19166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:27.083583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19105
2026-06-19T01:04:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2702 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:27 dist-test-slave-3rxd krb5kdc[19634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:27 dist-test-slave-3rxd krb5kdc[19634](info): set up 2 sockets
Jun 19 01:04:27 dist-test-slave-3rxd krb5kdc[19634](info): commencing operation
krb5kdc: starting...
W20260619 01:04:28.166993 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:28 dist-test-slave-3rxd krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:04:28.389264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45045
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:28.704866 19647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:28.705184 19647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:28.705353 19647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:28.712455 19647 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:28.712592 19647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:28.712697 19647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:28.712800 19647 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:28.712885 19647 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:28.722316 19647 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:28.723601 19647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:28.724906 19647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:28.736140 19652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:28.736245 19653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:28.736243 19655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:28.737222 19647 server_base.cc:1034] running on GCE node
I20260619 01:04:28.738039 19647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:28.739485 19647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:28.740679 19647 hybrid_clock.cc:648] HybridClock initialized: now 1781831068740644 us; error 32 us; skew 500 ppm
I20260619 01:04:28.741011 19647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:28 dist-test-slave-3rxd krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:28.743945 19647 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:04:28.745366 19647 webserver.cc:460] Webserver started at http://127.0.97.62:44047/ using document root <none> and password file <none>
I20260619 01:04:28.746028 19647 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:28.746125 19647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:28.746490 19647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:28.749322 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "ebd5327af589400a89a18f43900e7169"
format_stamp: "Formatted at 2026-06-19 01:04:28 on dist-test-slave-3rxd"
server_key: "87a29bb1415d701e9cbac1eca1b4e71d"
server_key_iv: "70ddd8c8f2ae6ffccd56710384b276f5"
server_key_version: "encryptionkey@0"
I20260619 01:04:28.750038 19647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "ebd5327af589400a89a18f43900e7169"
format_stamp: "Formatted at 2026-06-19 01:04:28 on dist-test-slave-3rxd"
server_key: "87a29bb1415d701e9cbac1eca1b4e71d"
server_key_iv: "70ddd8c8f2ae6ffccd56710384b276f5"
server_key_version: "encryptionkey@0"
I20260619 01:04:28.754855 19647 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260619 01:04:28.758031 19662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:28.759361 19647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:04:28.759547 19647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "ebd5327af589400a89a18f43900e7169"
format_stamp: "Formatted at 2026-06-19 01:04:28 on dist-test-slave-3rxd"
server_key: "87a29bb1415d701e9cbac1eca1b4e71d"
server_key_iv: "70ddd8c8f2ae6ffccd56710384b276f5"
server_key_version: "encryptionkey@0"
I20260619 01:04:28.759735 19647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:28.785259 19647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:28.789458 19647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:28.789783 19647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:28.808190 19647 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45045
I20260619 01:04:28.808238 19713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45045 every 8 connection(s)
I20260619 01:04:28.809495 19647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:28.809978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19647
I20260619 01:04:28.810177 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:28.810873 388 external_mini_cluster.cc:1428] Setting key ad88b19b6b775a34b690ebc68b9ecd37
I20260619 01:04:28.814388 19714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 01:04:28 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:28.826880 19714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: Bootstrap starting.
I20260619 01:04:28.831243 19714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:28.832599 19714 log.cc:824] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:28.835150 19717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:28.812627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.0.97.62:45045)
0619 01:04:28.813441 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:28.813456 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:28.813525 (+ 69us) server_negotiation.cc:407] Connection header received
0619 01:04:28.815950 (+ 2425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:28.815975 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:28.816456 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:28.816885 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:28.818064 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:28.820161 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:28.821463 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:28.822511 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:28.822710 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:28.826539 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:28.826574 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:28.826594 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:28.826649 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:28.830667 (+ 4018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:28.831363 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:28.831373 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:28.831381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:28.831606 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:28.832005 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:28.832011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:28.832014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:28.832726 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:28.833001 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:28.833419 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 01:04:28.833839 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":309,"threads_started":1}
I20260619 01:04:28.835994 19714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: No bootstrap required, opened a new log
I20260619 01:04:28.843617 19714 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260619 01:04:28.843922 19714 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:28.843993 19714 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebd5327af589400a89a18f43900e7169, State: Initialized, Role: FOLLOWER
I20260619 01:04:28.844458 19714 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260619 01:04:28.845024 19714 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:28.845115 19714 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:28.845229 19714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:28.847244 19714 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260619 01:04:28.847653 19714 leader_election.cc:302] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebd5327af589400a89a18f43900e7169; no voters:
I20260619 01:04:28.848167 19714 leader_election.cc:288] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:28.848307 19719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:28.849418 19719 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [term 1 LEADER]: Becoming Leader. State: Replica: ebd5327af589400a89a18f43900e7169, State: Running, Role: LEADER
I20260619 01:04:28.849936 19719 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } }
I20260619 01:04:28.850153 19714 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:28.853734 19720 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebd5327af589400a89a18f43900e7169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } } }
I20260619 01:04:28.853725 19721 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebd5327af589400a89a18f43900e7169. Latest consensus state: current_term: 1 leader_uuid: "ebd5327af589400a89a18f43900e7169" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebd5327af589400a89a18f43900e7169" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45045 } } }
I20260619 01:04:28.854317 19721 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:28.854314 19720 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:28.855338 19730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:28.860414 19730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:28.869197 19730 catalog_manager.cc:1347] Generated new cluster ID: 6f65215d64d148c7a250bbae5356ea90
I20260619 01:04:28.869300 19730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:28.896262 19730 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:28.898072 19730 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:28.908649 19730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: Generated new TSK 0
I20260619 01:04:28.909618 19730 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260619 01:04:28.980170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:29.300477 19740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:29.300727 19740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:29.300835 19740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:29.307966 19740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:29.308064 19740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:29.308254 19740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:29.317595 19740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:29.318828 19740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:29.320099 19740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:29.332877 19745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:29.332774 19746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:29.332976 19748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:29.333767 19740 server_base.cc:1034] running on GCE node
I20260619 01:04:29.334340 19740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:29.335314 19740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:29.336541 19740 hybrid_clock.cc:648] HybridClock initialized: now 1781831069336478 us; error 60 us; skew 500 ppm
I20260619 01:04:29.336839 19740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:29 dist-test-slave-3rxd krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:29.339823 19740 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260619 01:04:29.341179 19740 webserver.cc:460] Webserver started at http://127.0.97.1:36431/ using document root <none> and password file <none>
I20260619 01:04:29.341823 19740 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:29.341974 19740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:29.342279 19740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:29.345067 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c72cca8aea546349cd644650228ab55"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "3d606c53eb3e7cc21e82993b0924fbe8"
server_key_iv: "e0034f29b4a806c39edffa84864cd58b"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.345854 19740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c72cca8aea546349cd644650228ab55"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "3d606c53eb3e7cc21e82993b0924fbe8"
server_key_iv: "e0034f29b4a806c39edffa84864cd58b"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.350865 19740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 01:04:29.354148 19755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.355466 19740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:04:29.355635 19740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "9c72cca8aea546349cd644650228ab55"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "3d606c53eb3e7cc21e82993b0924fbe8"
server_key_iv: "e0034f29b4a806c39edffa84864cd58b"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.355834 19740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:29.383996 19740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:29.389734 19740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:29.390043 19740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:29.391268 19740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:29.393204 19740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:29.393317 19740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.393450 19740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:29.393514 19740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.429046 19740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42521
I20260619 01:04:29.429100 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42521 every 8 connection(s)
I20260619 01:04:29.430382 19740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:29.437541 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19740
I20260619 01:04:29.437727 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:29.438148 388 external_mini_cluster.cc:1428] Setting key 174a4679c11456e834a8b311230ed1c2
Jun 19 01:04:29 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:29.454262 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:29.433023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38897 (local address 127.0.97.62:45045)
0619 01:04:29.433515 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:29.433521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:29.434760 (+ 1239us) server_negotiation.cc:407] Connection header received
0619 01:04:29.437124 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:29.437129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:29.437238 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:29.437363 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:29.439333 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.440351 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.442867 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.443243 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.443336 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:29.447620 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:29.447628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:29.447636 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:29.447681 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:29.450957 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:29.451881 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:29.451885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:29.451887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:29.452020 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:29.452502 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:29.452507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:29.452510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:29.452793 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:29.452961 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:29.453772 (+ 811us) server_negotiation.cc:299] Negotiation successful
0619 01:04:29.453997 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":275,"threads_started":1}
I20260619 01:04:29.455659 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:29.433593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45045 (local address 127.0.97.1:38897)
0619 01:04:29.434507 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:29.434582 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:29.436890 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:29.437663 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:29.437693 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:29.438396 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:29.439135 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:29.439158 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.440500 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.440505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:29.442732 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:29.442750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.443469 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.443478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:29.443790 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:29.444478 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:29.444515 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:29.447388 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:29.451090 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:29.451102 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:29.451120 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:29.451750 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:29.452127 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:29.452134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:29.452141 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:29.452368 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:29.452944 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:29.452956 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:29.453376 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 01:04:29.453680 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 01:04:29.454253 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":329,"threads_started":1}
I20260619 01:04:29.457226 19868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45045
I20260619 01:04:29.457533 19868 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:29.458221 19868 heartbeater.cc:511] Master 127.0.97.62:45045 requested a full tablet report, sending...
I20260619 01:04:29.460457 19679 ts_manager.cc:194] Registered new tserver with Master: 9c72cca8aea546349cd644650228ab55 (127.0.97.1:42521)
I20260619 01:04:29.461910 19679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:38897
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260619 01:04:29.493844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:29.813547 19874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:29.813845 19874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:29.814023 19874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:29.821038 19874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:29.821192 19874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:29.821376 19874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:29.830483 19874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:29.831720 19874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:29.832980 19874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:29.845319 19880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:29.845399 19879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:29.845567 19882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:29.846352 19874 server_base.cc:1034] running on GCE node
I20260619 01:04:29.847021 19874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:29.847787 19874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:29.848976 19874 hybrid_clock.cc:648] HybridClock initialized: now 1781831069848921 us; error 46 us; skew 500 ppm
I20260619 01:04:29.849273 19874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:29 dist-test-slave-3rxd krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:29.858103 19874 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260619 01:04:29.859683 19874 webserver.cc:460] Webserver started at http://127.0.97.2:43033/ using document root <none> and password file <none>
I20260619 01:04:29.860322 19874 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:29.860451 19874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:29.860702 19874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:29.863736 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "7d0f8b53d40e422981a0847de1d9ceab"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "f5e6627d52cfdf590084e4dfae1392fa"
server_key_iv: "d64a1b20c6868a2fe2a9b410df8e7649"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.864691 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7d0f8b53d40e422981a0847de1d9ceab"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "f5e6627d52cfdf590084e4dfae1392fa"
server_key_iv: "d64a1b20c6868a2fe2a9b410df8e7649"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.869772 19874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 01:04:29.873155 19889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.874401 19874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:04:29.874588 19874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "7d0f8b53d40e422981a0847de1d9ceab"
format_stamp: "Formatted at 2026-06-19 01:04:29 on dist-test-slave-3rxd"
server_key: "f5e6627d52cfdf590084e4dfae1392fa"
server_key_iv: "d64a1b20c6868a2fe2a9b410df8e7649"
server_key_version: "encryptionkey@0"
I20260619 01:04:29.874802 19874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:29.893814 19874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:29.897424 19874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:29.897727 19874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:29.898602 19874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:29.899823 19874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:29.899924 19874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.900067 19874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:29.900136 19874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:29.931830 19874 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35087
I20260619 01:04:29.931903 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35087 every 8 connection(s)
I20260619 01:04:29.932953 19874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:29.934979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 19874
I20260619 01:04:29.935156 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:29.935704 388 external_mini_cluster.cc:1428] Setting key dfcc485778e5f5732aaecef58439b8d0
Jun 19 01:04:29 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:29.954643 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:29.935309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33149 (local address 127.0.97.62:45045)
0619 01:04:29.935522 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:29.935529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:29.936776 (+ 1247us) server_negotiation.cc:407] Connection header received
0619 01:04:29.939014 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:29.939018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:29.939075 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:29.939149 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:29.940969 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.941577 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.943790 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.944195 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.944307 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:29.948717 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:29.948721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:29.948724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:29.948755 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:29.951500 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:29.952337 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:29.952341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:29.952343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:29.952474 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:29.952946 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:29.952949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:29.952951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:29.953261 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:29.953378 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:29.954217 (+ 839us) server_negotiation.cc:299] Negotiation successful
0619 01:04:29.954397 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:04:29.956094 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:29.935746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45045 (local address 127.0.97.2:33149)
0619 01:04:29.936601 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:29.936658 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:29.938751 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:29.939391 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:29.939404 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:29.940112 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:29.940820 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:29.940845 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.941723 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.941730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:29.943658 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:29.943676 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:29.944435 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:29.944444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:29.944756 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:29.945429 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:29.945467 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:29.948487 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:29.951619 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:29.951630 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:29.951646 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:29.952224 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:29.952583 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:29.952591 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:29.952597 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:29.952833 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:29.953381 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:29.953391 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:29.953804 (+ 413us) client_negotiation.cc:769] Sending connection context
0619 01:04:29.954091 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 01:04:29.954643 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":298,"threads_started":1}
I20260619 01:04:29.957522 20002 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45045
I20260619 01:04:29.957831 20002 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:29.958549 20002 heartbeater.cc:511] Master 127.0.97.62:45045 requested a full tablet report, sending...
I20260619 01:04:29.960132 19679 ts_manager.cc:194] Registered new tserver with Master: 7d0f8b53d40e422981a0847de1d9ceab (127.0.97.2:35087)
I20260619 01:04:29.960911 19679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33149
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260619 01:04:29.991670 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:30.324231 20007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:30.324488 20007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:30.324580 20007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:30.331758 20007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:30.331852 20007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:30.331969 20007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:30.341054 20007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45045
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:30.342103 20007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:30.343386 20007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:30.356329 20012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:30.356251 20013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:30.356307 20015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:30.356823 20007 server_base.cc:1034] running on GCE node
I20260619 01:04:30.357422 20007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:30.358238 20007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:30.359427 20007 hybrid_clock.cc:648] HybridClock initialized: now 1781831070359383 us; error 37 us; skew 500 ppm
I20260619 01:04:30.359719 20007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:30 dist-test-slave-3rxd krb5kdc[19634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:30.362807 20007 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260619 01:04:30.364329 20007 webserver.cc:460] Webserver started at http://127.0.97.3:36035/ using document root <none> and password file <none>
I20260619 01:04:30.365000 20007 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:30.365155 20007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:30.365461 20007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:30.368248 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "e2ff09d399ec482a966924159848957f"
format_stamp: "Formatted at 2026-06-19 01:04:30 on dist-test-slave-3rxd"
server_key: "709c60e5e9c7587cb8ff23094c4a0ff8"
server_key_iv: "292a06b33c07b6752ec05faae6bc3dc0"
server_key_version: "encryptionkey@0"
I20260619 01:04:30.369015 20007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e2ff09d399ec482a966924159848957f"
format_stamp: "Formatted at 2026-06-19 01:04:30 on dist-test-slave-3rxd"
server_key: "709c60e5e9c7587cb8ff23094c4a0ff8"
server_key_iv: "292a06b33c07b6752ec05faae6bc3dc0"
server_key_version: "encryptionkey@0"
I20260619 01:04:30.373831 20007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:04:30.377019 20022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:30.378286 20007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 01:04:30.378455 20007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "e2ff09d399ec482a966924159848957f"
format_stamp: "Formatted at 2026-06-19 01:04:30 on dist-test-slave-3rxd"
server_key: "709c60e5e9c7587cb8ff23094c4a0ff8"
server_key_iv: "292a06b33c07b6752ec05faae6bc3dc0"
server_key_version: "encryptionkey@0"
I20260619 01:04:30.378640 20007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:30.401219 20007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:30.404923 20007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:30.405162 20007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:30.406013 20007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:30.407219 20007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:30.407287 20007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:30.407366 20007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:30.407407 20007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:30.439594 20007 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33539
I20260619 01:04:30.439671 20134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33539 every 8 connection(s)
I20260619 01:04:30.440768 20007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:30.442240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20007
I20260619 01:04:30.442433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:30.442986 388 external_mini_cluster.cc:1428] Setting key 5ab64acfc3ed725692d50923666025d2
Jun 19 01:04:30 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:30.460587 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.443309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50251 (local address 127.0.97.62:45045)
0619 01:04:30.443532 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.443538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.444845 (+ 1307us) server_negotiation.cc:407] Connection header received
0619 01:04:30.446997 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.447020 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.447101 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.447218 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:30.448714 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.449437 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.451088 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.451506 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.451641 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.454949 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:30.454954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:30.454957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:30.454995 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:30.457783 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.458462 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.458465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.458467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.458585 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.459077 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.459080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.459082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.459320 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:30.459446 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.460152 (+ 706us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.460317 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 01:04:30.461432 20137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.443793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45045 (local address 127.0.97.3:50251)
0619 01:04:30.444673 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.444731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.446676 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.447465 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.447477 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.448050 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:30.448560 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.448575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.449582 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.449587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.450943 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.450954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.451720 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.451727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.451988 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.452608 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:04:30.452628 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:30.454760 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:30.457907 (+ 3147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:30.457916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:30.457935 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:30.458347 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:30.458675 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:04:30.458678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:04:30.458680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:04:30.458921 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:04:30.459448 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:30.459453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:30.459796 (+ 343us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.460007 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.460406 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":425,"threads_started":1}
I20260619 01:04:30.462584 20135 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45045
I20260619 01:04:30.462888 20135 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:30.463569 20135 heartbeater.cc:511] Master 127.0.97.62:45045 requested a full tablet report, sending...
I20260619 01:04:30.464738 19868 heartbeater.cc:503] Master 127.0.97.62:45045 was elected leader, sending a full tablet report...
I20260619 01:04:30.464892 19679 ts_manager.cc:194] Registered new tserver with Master: e2ff09d399ec482a966924159848957f (127.0.97.3:33539)
I20260619 01:04:30.466014 19679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50251
I20260619 01:04:30.469839 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:30.485772 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.473799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.0.97.62:45045)
0619 01:04:30.473975 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.473980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.474289 (+ 309us) server_negotiation.cc:407] Connection header received
0619 01:04:30.474487 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.474491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.474545 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.474641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:30.476034 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.476701 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.478063 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.478443 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.478577 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.480828 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:30.480834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:30.480838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:30.480902 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:30.483587 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.484181 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.484184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.484187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.484290 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.484578 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.484581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.484583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.484849 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:30.485036 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.485280 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.485445 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:04:30.489140 19679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:30.495664 19679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:30.528046 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.507585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51770 (local address 127.0.97.2:35087)
0619 01:04:30.518581 (+ 10996us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.518594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.518628 (+ 34us) server_negotiation.cc:407] Connection header received
0619 01:04:30.518774 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.518786 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.519109 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.520152 (+ 1043us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:30.520410 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.522304 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.525016 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.526339 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.526487 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.526687 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.527572 (+ 885us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.527727 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10839,"thread_start_us":10711,"threads_started":1}
I20260619 01:04:30.528232 20147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.507451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35087 (local address 127.0.0.1:51770)
0619 01:04:30.517945 (+ 10494us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.517970 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.518075 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.519390 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.519401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.519458 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:30.519918 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.519933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.522449 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.522455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.524805 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.524826 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.526818 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.526828 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.527228 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.527372 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.527501 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.527721 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10349,"mutex_wait_us":237,"thread_start_us":9415,"threads_started":1}
I20260619 01:04:30.528347 20149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.507137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42521 (local address 127.0.0.1:46684)
0619 01:04:30.517274 (+ 10137us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.517319 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.517519 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.520060 (+ 2541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.520068 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.520103 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:30.520600 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.520614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.522857 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.522862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.524845 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.524855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.526856 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.526861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.527228 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.527372 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.527501 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.527721 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9870,"mutex_wait_us":185,"thread_start_us":10543,"threads_started":1}
I20260619 01:04:30.528790 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.507284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46684 (local address 127.0.97.1:42521)
0619 01:04:30.519088 (+ 11804us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.519096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.519122 (+ 26us) server_negotiation.cc:407] Connection header received
0619 01:04:30.519220 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.519231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.519535 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.519760 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:30.520838 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.522675 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.524986 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.526480 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.526601 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.526845 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.528303 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.528490 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11653,"thread_start_us":11485,"threads_started":1}
I20260619 01:04:30.531304 20152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.519098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33539 (local address 127.0.0.1:43382)
0619 01:04:30.522987 (+ 3889us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.523007 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.523121 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.523765 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.523769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.523793 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:30.524271 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.524282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.526152 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.526157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.528017 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.528035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.530488 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.530494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.530676 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.530698 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.530847 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.530963 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3759,"mutex_wait_us":1,"thread_start_us":3560,"threads_started":1}
I20260619 01:04:30.531364 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.518417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.97.3:33539)
0619 01:04:30.522420 (+ 4003us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.522427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.523247 (+ 820us) server_negotiation.cc:407] Connection header received
0619 01:04:30.523345 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.523353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.523575 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.523752 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:30.524459 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.525986 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.528684 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.530025 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.530150 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.530352 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.530887 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.531056 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3849,"thread_start_us":3708,"threads_started":1}
I20260619 01:04:30.532126 19937 tablet_service.cc:1467] Processing CreateTablet for tablet a866ceb935064713857768da2af124a9 (DEFAULT_TABLE table=test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:30.532126 19803 tablet_service.cc:1467] Processing CreateTablet for tablet a866ceb935064713857768da2af124a9 (DEFAULT_TABLE table=test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:30.533514 19803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a866ceb935064713857768da2af124a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:30.533538 19937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a866ceb935064713857768da2af124a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:30.534319 20070 tablet_service.cc:1467] Processing CreateTablet for tablet a866ceb935064713857768da2af124a9 (DEFAULT_TABLE table=test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:30.535554 20070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a866ceb935064713857768da2af124a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:30.546325 20153 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Bootstrap starting.
I20260619 01:04:30.549675 20153 tablet_bootstrap.cc:654] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:30.551138 20153 log.cc:824] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:30.553390 20154 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Bootstrap starting.
I20260619 01:04:30.554239 20155 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Bootstrap starting.
I20260619 01:04:30.557415 20153 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: No bootstrap required, opened a new log
I20260619 01:04:30.557631 20154 tablet_bootstrap.cc:654] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:30.557637 20155 tablet_bootstrap.cc:654] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:30.557709 20153 ts_tablet_manager.cc:1397] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260619 01:04:30.558869 20155 log.cc:824] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:30.558871 20154 log.cc:824] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:30.562048 20154 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: No bootstrap required, opened a new log
I20260619 01:04:30.562340 20154 ts_tablet_manager.cc:1397] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 01:04:30.565826 20153 raft_consensus.cc:348] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.566169 20153 raft_consensus.cc:374] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:30.566288 20153 raft_consensus.cc:729] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2ff09d399ec482a966924159848957f, State: Initialized, Role: FOLLOWER
I20260619 01:04:30.566946 20153 consensus_queue.cc:260] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.567903 20155 tablet_bootstrap.cc:492] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: No bootstrap required, opened a new log
I20260619 01:04:30.568163 20155 ts_tablet_manager.cc:1397] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260619 01:04:30.571990 20135 heartbeater.cc:503] Master 127.0.97.62:45045 was elected leader, sending a full tablet report...
I20260619 01:04:30.573019 20153 ts_tablet_manager.cc:1428] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260619 01:04:30.573843 20154 raft_consensus.cc:348] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.574235 20154 raft_consensus.cc:374] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:30.574334 20154 raft_consensus.cc:729] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c72cca8aea546349cd644650228ab55, State: Initialized, Role: FOLLOWER
I20260619 01:04:30.574986 20154 consensus_queue.cc:260] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.576660 20154 ts_tablet_manager.cc:1428] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260619 01:04:30.578223 20155 raft_consensus.cc:348] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.578667 20155 raft_consensus.cc:374] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:30.578794 20155 raft_consensus.cc:729] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0f8b53d40e422981a0847de1d9ceab, State: Initialized, Role: FOLLOWER
I20260619 01:04:30.579278 20155 consensus_queue.cc:260] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.580371 20002 heartbeater.cc:503] Master 127.0.97.62:45045 was elected leader, sending a full tablet report...
I20260619 01:04:30.580850 20155 ts_tablet_manager.cc:1428] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 01:04:30.618832 20161 raft_consensus.cc:482] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:04:30.619199 20161 raft_consensus.cc:504] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.620777 20161 leader_election.cc:288] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c72cca8aea546349cd644650228ab55 (127.0.97.1:42521), e2ff09d399ec482a966924159848957f (127.0.97.3:33539)
I20260619 01:04:30.628226 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.621173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56639 (local address 127.0.97.1:42521)
0619 01:04:30.621354 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.621358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.621725 (+ 367us) server_negotiation.cc:407] Connection header received
0619 01:04:30.622089 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.622095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.622172 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.622241 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:30.622911 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.623849 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.626309 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.627408 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.627545 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.627644 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.627855 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.627996 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:04:30.628320 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.620996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42521 (local address 127.0.97.2:56639)
0619 01:04:30.621627 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.621651 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.621851 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.622391 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.622394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.622412 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:30.622767 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.622781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.624003 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.624007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.626145 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.626164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.627600 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.627604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.627710 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.627733 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.627805 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.627887 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":27,"thread_start_us":310,"threads_started":1}
I20260619 01:04:30.628912 20163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.621702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33539 (local address 127.0.97.2:35007)
0619 01:04:30.622294 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:30.622307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:30.622504 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:30.622935 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:30.622938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:30.622955 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:30.623302 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.623310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.624611 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.624621 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.626579 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:30.626588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.628124 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.628130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:30.628345 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.628399 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 01:04:30.628504 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 01:04:30.628664 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"mutex_wait_us":4,"spinlock_wait_cycles":21760,"thread_start_us":316,"threads_started":1}
I20260619 01:04:30.629302 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.621945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35007 (local address 127.0.97.3:33539)
0619 01:04:30.622149 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.622157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.622389 (+ 232us) server_negotiation.cc:407] Connection header received
0619 01:04:30.622667 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.622672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.622798 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.622898 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:30.623460 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.624469 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.626738 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.627920 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.628035 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.628125 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.628902 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.629045 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 01:04:30.629336 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a866ceb935064713857768da2af124a9" candidate_uuid: "7d0f8b53d40e422981a0847de1d9ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c72cca8aea546349cd644650228ab55" is_pre_election: true
I20260619 01:04:30.629899 19823 raft_consensus.cc:2454] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0f8b53d40e422981a0847de1d9ceab in term 0.
I20260619 01:04:30.630242 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a866ceb935064713857768da2af124a9" candidate_uuid: "7d0f8b53d40e422981a0847de1d9ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ff09d399ec482a966924159848957f" is_pre_election: true
I20260619 01:04:30.630621 19893 leader_election.cc:302] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0f8b53d40e422981a0847de1d9ceab, 9c72cca8aea546349cd644650228ab55; no voters:
I20260619 01:04:30.630865 20090 raft_consensus.cc:2454] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0f8b53d40e422981a0847de1d9ceab in term 0.
I20260619 01:04:30.631080 20161 raft_consensus.cc:2781] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:30.631235 20161 raft_consensus.cc:482] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:30.631351 20161 raft_consensus.cc:3037] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:30.633510 20161 raft_consensus.cc:504] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.634079 20161 leader_election.cc:288] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [CANDIDATE]: Term 1 election: Requested vote from peers 9c72cca8aea546349cd644650228ab55 (127.0.97.1:42521), e2ff09d399ec482a966924159848957f (127.0.97.3:33539)
I20260619 01:04:30.634445 19823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a866ceb935064713857768da2af124a9" candidate_uuid: "7d0f8b53d40e422981a0847de1d9ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c72cca8aea546349cd644650228ab55"
I20260619 01:04:30.634533 20090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a866ceb935064713857768da2af124a9" candidate_uuid: "7d0f8b53d40e422981a0847de1d9ceab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2ff09d399ec482a966924159848957f"
I20260619 01:04:30.634635 19823 raft_consensus.cc:3037] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:30.634775 20090 raft_consensus.cc:3037] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:30.636782 19823 raft_consensus.cc:2454] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0f8b53d40e422981a0847de1d9ceab in term 1.
I20260619 01:04:30.636927 20090 raft_consensus.cc:2454] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0f8b53d40e422981a0847de1d9ceab in term 1.
I20260619 01:04:30.637204 19893 leader_election.cc:302] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0f8b53d40e422981a0847de1d9ceab, 9c72cca8aea546349cd644650228ab55; no voters:
I20260619 01:04:30.637480 20161 raft_consensus.cc:2781] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:30.637914 20161 raft_consensus.cc:686] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 1 LEADER]: Becoming Leader. State: Replica: 7d0f8b53d40e422981a0847de1d9ceab, State: Running, Role: LEADER
I20260619 01:04:30.638345 20161 consensus_queue.cc:237] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } }
I20260619 01:04:30.642647 19679 catalog_manager.cc:5495] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d0f8b53d40e422981a0847de1d9ceab (127.0.97.2). New cstate: current_term: 1 leader_uuid: "7d0f8b53d40e422981a0847de1d9ceab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d0f8b53d40e422981a0847de1d9ceab" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 } health_report { overall_health: UNKNOWN } } }
W20260619 01:04:30.682967 19869 tablet.cc:2369] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:30.684629 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.679036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.0.97.2:35087)
0619 01:04:30.679226 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.679233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.679317 (+ 84us) server_negotiation.cc:407] Connection header received
0619 01:04:30.679565 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.679569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.679641 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.679790 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:30.680152 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.680846 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.682285 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.682773 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.682900 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.683480 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:30.684057 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:30.684169 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.684240 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.684299 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:04:30.684665 20003 tablet.cc:2369] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:30.692116 19823 raft_consensus.cc:1261] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 1 FOLLOWER]: Refusing update from remote peer 7d0f8b53d40e422981a0847de1d9ceab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:30.692070 20090 raft_consensus.cc:1261] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 1 FOLLOWER]: Refusing update from remote peer 7d0f8b53d40e422981a0847de1d9ceab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 01:04:30.692313 20136 tablet.cc:2369] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:30.693106 20164 consensus_queue.cc:1035] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c72cca8aea546349cd644650228ab55" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:30.693424 20161 consensus_queue.cc:1035] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2ff09d399ec482a966924159848957f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:30.703926 20167 mvcc.cc:205] Tried to move back new op lower bound from 7298380065545265152 to 7298380065338814464. Current Snapshot: MvccSnapshot[applied={T|T < 7298380065545265152}]
I20260619 01:04:30.716390 19679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:30.716660 19679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:30.720126 19679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ebd5327af589400a89a18f43900e7169: Sending DeleteTablet for 3 replicas of tablet a866ceb935064713857768da2af124a9
I20260619 01:04:30.721199 19937 tablet_service.cc:1514] Processing DeleteTablet for tablet a866ceb935064713857768da2af124a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51770
I20260619 01:04:30.721313 19803 tablet_service.cc:1514] Processing DeleteTablet for tablet a866ceb935064713857768da2af124a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46684
I20260619 01:04:30.721346 20070 tablet_service.cc:1514] Processing DeleteTablet for tablet a866ceb935064713857768da2af124a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:30 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260619 01:04:30.723022 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19740
I20260619 01:04:30.733322 20178 tablet_replica.cc:330] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: stopping tablet replica
I20260619 01:04:30.733543 20177 tablet_replica.cc:330] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: stopping tablet replica
I20260619 01:04:30.733899 20177 raft_consensus.cc:2229] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:30.733902 20178 raft_consensus.cc:2229] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:30.734407 20177 raft_consensus.cc:2258] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:30.734822 20178 raft_consensus.cc:2258] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:30.735832 20176 tablet_replica.cc:330] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: stopping tablet replica
I20260619 01:04:30.736245 20176 raft_consensus.cc:2229] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:30.736868 20176 raft_consensus.cc:2258] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:30.737156 20178 ts_tablet_manager.cc:1905] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:30.738893 20176 ts_tablet_manager.cc:1905] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:30.744194 20178 ts_tablet_manager.cc:1918] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:30.744385 20178 log.cc:1196] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/a866ceb935064713857768da2af124a9
I20260619 01:04:30.744951 20178 ts_tablet_manager.cc:1939] T a866ceb935064713857768da2af124a9 P 9c72cca8aea546349cd644650228ab55: Deleting consensus metadata
Jun 19 01:04:30 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260619 01:04:30.746250 20176 ts_tablet_manager.cc:1918] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:30.746438 20176 log.cc:1196] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/a866ceb935064713857768da2af124a9
I20260619 01:04:30.746917 19666 catalog_manager.cc:4841] TS 9c72cca8aea546349cd644650228ab55 (127.0.97.1:42521): tablet a866ceb935064713857768da2af124a9 (table test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]) successfully deleted
I20260619 01:04:30.747053 20176 ts_tablet_manager.cc:1939] T a866ceb935064713857768da2af124a9 P 7d0f8b53d40e422981a0847de1d9ceab: Deleting consensus metadata
I20260619 01:04:30.748924 20177 ts_tablet_manager.cc:1905] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:30.748924 19665 catalog_manager.cc:4841] TS 7d0f8b53d40e422981a0847de1d9ceab (127.0.97.2:35087): tablet a866ceb935064713857768da2af124a9 (table test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]) successfully deleted
I20260619 01:04:30.754127 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.724096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.0.97.1:42521)
0619 01:04:30.724256 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.724261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.735213 (+ 10952us) server_negotiation.cc:407] Connection header received
0619 01:04:30.735416 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.735421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.735512 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.735611 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:30.737290 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.738475 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.740376 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.741028 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.741131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.746834 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:30.746847 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:30.746863 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:30.746911 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:30.750832 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.751747 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.751756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.751760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.751920 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.752339 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.752346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.752353 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.752793 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:30.753078 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.753387 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.753605 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 01:04:30.754794 19863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:30.755599 20177 ts_tablet_manager.cc:1918] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:30.755792 20177 log.cc:1196] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/a866ceb935064713857768da2af124a9
I20260619 01:04:30.756351 20177 ts_tablet_manager.cc:1939] T a866ceb935064713857768da2af124a9 P e2ff09d399ec482a966924159848957f: Deleting consensus metadata
I20260619 01:04:30.758188 19666 catalog_manager.cc:4841] TS e2ff09d399ec482a966924159848957f (127.0.97.3:33539): tablet a866ceb935064713857768da2af124a9 (table test-table [id=fba6dc02b9d44f47a1d902b034f7ba9a]) successfully deleted
I20260619 01:04:30.843442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19740
I20260619 01:04:30.862314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19874
Jun 19 01:04:30 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260619 01:04:30.879411 20150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:30.863156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.0.97.2:35087)
0619 01:04:30.863375 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:30.863381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:30.863403 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:04:30.863529 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:30.863535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:30.863629 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:30.863735 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:30.865004 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.866230 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.867804 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:30.868486 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:30.868620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:30.872661 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:30.872681 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:30.872702 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:30.872748 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:30.876597 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.877188 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.877198 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.877202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.877377 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:30.877749 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:30.877757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:30.877760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:30.878206 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:30.878595 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:30.878779 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 01:04:30.879002 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:04:30.880088 19997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:30.988725 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19874
I20260619 01:04:31.007556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20007
Jun 19 01:04:31 dist-test-slave-3rxd krb5kdc[19634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260619 01:04:31.022250 20151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:31.008355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.97.3:33539)
0619 01:04:31.008538 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:31.008545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:31.008582 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:04:31.008789 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:31.008799 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:31.008861 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:31.008981 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:31.010195 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:31.011031 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:31.012331 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:31.012738 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:31.012863 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:31.017026 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:31.017036 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:31.017051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:31.017097 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:31.019815 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:31.020512 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:31.020518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:31.020521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:31.020652 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:31.020958 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:31.020961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:31.020963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:31.021276 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:31.021517 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:31.021761 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 01:04:31.021913 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:04:31.022897 20130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:31.110749 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20007
I20260619 01:04:31.126891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 19647
I20260619 01:04:31.127957 19709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:31.235930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 19647
2026-06-19T01:04:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4153 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T01:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:31Z Disabled control of system clock
I20260619 01:04:31.309376 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:31.626495 20187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:31.626776 20187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:31.626941 20187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:31.633879 20187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:31.634003 20187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:31.634074 20187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:31.634140 20187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:31.634227 20187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:31.642954 20187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:31.644089 20187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:31.645773 20187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:31.657297 20192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:31.657255 20193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:31.657471 20195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:31.657963 20187 server_base.cc:1034] running on GCE node
I20260619 01:04:31.658800 20187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:31.660111 20187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:31.661343 20187 hybrid_clock.cc:648] HybridClock initialized: now 1781831071661309 us; error 48 us; skew 500 ppm
I20260619 01:04:31.661710 20187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:31.664021 20187 webserver.cc:460] Webserver started at http://127.0.97.62:40939/ using document root <none> and password file <none>
I20260619 01:04:31.664682 20187 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:31.664817 20187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:31.665136 20187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:31.667994 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "e5b5c4c337924ed587233408c527be5f"
format_stamp: "Formatted at 2026-06-19 01:04:31 on dist-test-slave-3rxd"
server_key: "e14f7dce9e09d6ef3b9633e5d86317d8"
server_key_iv: "0c8a44286b220be77a7a6dcef58278f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:31.668763 20187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "e5b5c4c337924ed587233408c527be5f"
format_stamp: "Formatted at 2026-06-19 01:04:31 on dist-test-slave-3rxd"
server_key: "e14f7dce9e09d6ef3b9633e5d86317d8"
server_key_iv: "0c8a44286b220be77a7a6dcef58278f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:31.673600 20187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 01:04:31.676846 20201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:31.678040 20187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 01:04:31.678215 20187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "e5b5c4c337924ed587233408c527be5f"
format_stamp: "Formatted at 2026-06-19 01:04:31 on dist-test-slave-3rxd"
server_key: "e14f7dce9e09d6ef3b9633e5d86317d8"
server_key_iv: "0c8a44286b220be77a7a6dcef58278f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:31.678435 20187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:31.694687 20187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:31.695498 20187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:31.695765 20187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:31.713536 20187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41237
I20260619 01:04:31.713546 20252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41237 every 8 connection(s)
I20260619 01:04:31.714788 20187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:31.718302 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:31.718819 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20187
I20260619 01:04:31.718987 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:31.719410 388 external_mini_cluster.cc:1428] Setting key cb6557e4b423fcc511bc19cff2493df2
I20260619 01:04:31.729893 20253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: Bootstrap starting.
I20260619 01:04:31.732195 20256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:31.721163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.0.97.62:41237)
0619 01:04:31.722118 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:31.722128 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:31.722166 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:04:31.724175 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:31.724196 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:31.724628 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:31.725006 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:31.725525 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:31.726775 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:31.728054 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:31.728686 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:31.728883 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:31.730083 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:31.730091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:31.730104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:31.730615 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:31.730684 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:31.730703 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:31.730891 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:31.731061 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:04:31.731362 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":345,"threads_started":1}
I20260619 01:04:31.732999 20253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:31.734251 20253 log.cc:824] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:31.736850 20253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: No bootstrap required, opened a new log
I20260619 01:04:31.744220 20253 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260619 01:04:31.744601 20253 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:31.744678 20253 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b5c4c337924ed587233408c527be5f, State: Initialized, Role: FOLLOWER
I20260619 01:04:31.745121 20253 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260619 01:04:31.745306 20253 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:31.745396 20253 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:31.745507 20253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:31.747515 20253 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260619 01:04:31.747960 20253 leader_election.cc:302] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5b5c4c337924ed587233408c527be5f; no voters:
I20260619 01:04:31.748445 20253 leader_election.cc:288] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:31.748588 20258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:31.749267 20258 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [term 1 LEADER]: Becoming Leader. State: Replica: e5b5c4c337924ed587233408c527be5f, State: Running, Role: LEADER
I20260619 01:04:31.749770 20258 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } }
I20260619 01:04:31.750280 20253 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:31.752955 20260 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5b5c4c337924ed587233408c527be5f. Latest consensus state: current_term: 1 leader_uuid: "e5b5c4c337924ed587233408c527be5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260619 01:04:31.753273 20260 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:31.752753 20259 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5b5c4c337924ed587233408c527be5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5b5c4c337924ed587233408c527be5f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41237 } } }
I20260619 01:04:31.753886 20259 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:31.754835 20265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:31.759744 20265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:31.767886 20265 catalog_manager.cc:1347] Generated new cluster ID: 140fcfbc19b94add89a363413a105a37
I20260619 01:04:31.767998 20265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:31.791662 20265 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:31.792918 20265 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:31.799443 20265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: Generated new TSK 0
I20260619 01:04:31.800209 20265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:31.804770 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 01:04:32.132843 20277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:32.133083 20277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:32.133162 20277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:32.139981 20277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:32.140069 20277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:32.140165 20277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260619 01:04:32.148772 20277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:32.149957 20277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:32.152002 20277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:32.165102 20282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:32.165130 20283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:32.165069 20285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:32.165928 20277 server_base.cc:1034] running on GCE node
I20260619 01:04:32.166582 20277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:32.167515 20277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:32.168756 20277 hybrid_clock.cc:648] HybridClock initialized: now 1781831072168710 us; error 55 us; skew 500 ppm
I20260619 01:04:32.169080 20277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:32.171329 20277 webserver.cc:460] Webserver started at http://127.0.97.1:43205/ using document root <none> and password file <none>
I20260619 01:04:32.172016 20277 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:32.172147 20277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:32.172506 20277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:32.175393 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data/instance:
uuid: "619bd4c1018442fdaf606163f0e97be3"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "2744c07a83c535c06dee4733cdfe08f0"
server_key_iv: "b49df8802f5411405e3b3d45a4a6bc11"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.176220 20277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal/instance:
uuid: "619bd4c1018442fdaf606163f0e97be3"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "2744c07a83c535c06dee4733cdfe08f0"
server_key_iv: "b49df8802f5411405e3b3d45a4a6bc11"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.181154 20277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 01:04:32.184535 20291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.185701 20277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:04:32.185868 20277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal
uuid: "619bd4c1018442fdaf606163f0e97be3"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "2744c07a83c535c06dee4733cdfe08f0"
server_key_iv: "b49df8802f5411405e3b3d45a4a6bc11"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.186097 20277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:32.197438 20277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:32.198197 20277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:32.198474 20277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:32.199609 20277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:32.200910 20277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:32.201030 20277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.201157 20277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:32.201248 20277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.233258 20277 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38183
I20260619 01:04:32.233314 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38183 every 8 connection(s)
I20260619 01:04:32.234561 20277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/data/info.pb
I20260619 01:04:32.239952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20277
I20260619 01:04:32.240118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal/instance
I20260619 01:04:32.240684 388 external_mini_cluster.cc:1428] Setting key 0d6eea50a9ef1fea47c46d19e7d422da
I20260619 01:04:32.243579 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:32.248234 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:32.236993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59737 (local address 127.0.97.62:41237)
0619 01:04:32.237434 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:32.237445 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:32.238455 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 01:04:32.240572 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:32.240576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:32.240643 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:32.240736 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:32.241989 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.243018 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.245458 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.245846 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.245950 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:32.247529 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:32.247533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:32.247536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:32.247553 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:32.247596 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:32.247608 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:32.247673 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:32.247906 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 01:04:32.248007 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":245,"threads_started":1}
I20260619 01:04:32.249636 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:32.237438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.1:59737)
0619 01:04:32.238290 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:32.238330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:32.240271 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:32.241009 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:32.241020 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:32.241129 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:32.241824 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:32.241842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.243171 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.243176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:32.245299 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:32.245318 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.246085 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.246094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:32.246409 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:32.247181 (+ 772us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:32.247187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:32.247235 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:32.247338 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:32.247699 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:32.247712 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:32.247723 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 01:04:32.247890 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 01:04:32.248292 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260619 01:04:32.251358 20404 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260619 01:04:32.251673 20404 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:32.252557 20404 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260619 01:04:32.254776 20218 ts_manager.cc:194] Registered new tserver with Master: 619bd4c1018442fdaf606163f0e97be3 (127.0.97.1:38183)
I20260619 01:04:32.256986 20218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:59737
W20260619 01:04:32.562034 20408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:32.562278 20408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:32.562358 20408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:32.569201 20408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:32.569298 20408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:32.569398 20408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260619 01:04:32.578059 20408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:32.579262 20408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:32.580888 20408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:32.593374 20413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:32.593494 20416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:32.593523 20414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:32.594551 20408 server_base.cc:1034] running on GCE node
I20260619 01:04:32.595147 20408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:32.596022 20408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:32.597225 20408 hybrid_clock.cc:648] HybridClock initialized: now 1781831072597189 us; error 32 us; skew 500 ppm
I20260619 01:04:32.597532 20408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:32.599838 20408 webserver.cc:460] Webserver started at http://127.0.97.2:45651/ using document root <none> and password file <none>
I20260619 01:04:32.600467 20408 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:32.600582 20408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:32.600893 20408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:32.603636 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data/instance:
uuid: "99234f9760cf4e9084724725e0faf9bf"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "b6f0becf6e684c5db25d297992d56089"
server_key_iv: "c232b190dbcb869b37196043b1b841d7"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.604348 20408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal/instance:
uuid: "99234f9760cf4e9084724725e0faf9bf"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "b6f0becf6e684c5db25d297992d56089"
server_key_iv: "c232b190dbcb869b37196043b1b841d7"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.609095 20408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:32.612412 20422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.613647 20408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:04:32.613818 20408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal
uuid: "99234f9760cf4e9084724725e0faf9bf"
format_stamp: "Formatted at 2026-06-19 01:04:32 on dist-test-slave-3rxd"
server_key: "b6f0becf6e684c5db25d297992d56089"
server_key_iv: "c232b190dbcb869b37196043b1b841d7"
server_key_version: "encryptionkey@0"
I20260619 01:04:32.614028 20408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:32.628294 20408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:32.629017 20408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:32.629287 20408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:32.630196 20408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:32.631705 20408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:32.631830 20408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.631963 20408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:32.632052 20408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:32.664418 20408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45279
I20260619 01:04:32.664464 20534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45279 every 8 connection(s)
I20260619 01:04:32.665709 20408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/data/info.pb
I20260619 01:04:32.670677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20408
I20260619 01:04:32.670861 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal/instance
I20260619 01:04:32.671386 388 external_mini_cluster.cc:1428] Setting key 9cda94e54442667798770353b8ff4aa3
I20260619 01:04:32.674046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--builtin_ntp_servers=127.0.97.20:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 01:04:32.678548 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:32.668138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60177 (local address 127.0.97.62:41237)
0619 01:04:32.668347 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:32.668353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:32.669786 (+ 1433us) server_negotiation.cc:407] Connection header received
0619 01:04:32.672020 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:32.672026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:32.672119 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:32.672225 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:32.673407 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.674081 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.675875 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.676305 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.676449 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:32.677743 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:32.677748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:32.677751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:32.677767 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:32.677796 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:32.677807 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:32.677898 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:32.678120 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 01:04:32.678213 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 01:04:32.679539 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:32.668689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.2:60177)
0619 01:04:32.669562 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:32.669597 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:32.671704 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:32.672481 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:32.672489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:32.672572 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:32.673240 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:32.673265 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.674311 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.674315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:32.675727 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:32.675739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:32.676522 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:32.676529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:32.676761 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:32.677445 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:32.677447 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:32.677484 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:32.677550 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:32.677911 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:32.677937 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:32.677943 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 01:04:32.678106 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 01:04:32.678483 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":254,"threads_started":1}
I20260619 01:04:32.682181 20535 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260619 01:04:32.682572 20535 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:32.683398 20535 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260619 01:04:32.684795 20218 ts_manager.cc:194] Registered new tserver with Master: 99234f9760cf4e9084724725e0faf9bf (127.0.97.2:45279)
I20260619 01:04:32.685590 20218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:60177
W20260619 01:04:32.996158 20538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:32.996434 20538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:32.996579 20538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:33.003834 20538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:33.003949 20538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:33.004136 20538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260619 01:04:33.013370 20538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:33.014709 20538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:33.016536 20538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:33.029121 20544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:33.029254 20543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:33.029271 20546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:33.030025 20538 server_base.cc:1034] running on GCE node
I20260619 01:04:33.030665 20538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:33.031627 20538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:33.032884 20538 hybrid_clock.cc:648] HybridClock initialized: now 1781831073032841 us; error 51 us; skew 500 ppm
I20260619 01:04:33.033231 20538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 01:04:33.035470 20538 webserver.cc:460] Webserver started at http://127.0.97.3:41611/ using document root <none> and password file <none>
I20260619 01:04:33.036147 20538 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:33.036278 20538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:33.036602 20538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:33.039428 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data/instance:
uuid: "9d6931dcf8954aa3b0bab24adff35b4b"
format_stamp: "Formatted at 2026-06-19 01:04:33 on dist-test-slave-3rxd"
server_key: "bb01d899dd20fc9ce0346593b3142670"
server_key_iv: "3fcbddf684a9de30a963be2cc7c4e210"
server_key_version: "encryptionkey@0"
I20260619 01:04:33.040185 20538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9d6931dcf8954aa3b0bab24adff35b4b"
format_stamp: "Formatted at 2026-06-19 01:04:33 on dist-test-slave-3rxd"
server_key: "bb01d899dd20fc9ce0346593b3142670"
server_key_iv: "3fcbddf684a9de30a963be2cc7c4e210"
server_key_version: "encryptionkey@0"
I20260619 01:04:33.045823 20538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 01:04:33.049137 20552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:33.050343 20538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:04:33.050518 20538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal
uuid: "9d6931dcf8954aa3b0bab24adff35b4b"
format_stamp: "Formatted at 2026-06-19 01:04:33 on dist-test-slave-3rxd"
server_key: "bb01d899dd20fc9ce0346593b3142670"
server_key_iv: "3fcbddf684a9de30a963be2cc7c4e210"
server_key_version: "encryptionkey@0"
I20260619 01:04:33.050793 20538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:33.062130 20538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:33.062901 20538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:33.063153 20538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:33.064049 20538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 01:04:33.065394 20538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 01:04:33.065513 20538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:33.065650 20538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 01:04:33.065733 20538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:33.098037 20538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38011
I20260619 01:04:33.098091 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38011 every 8 connection(s)
I20260619 01:04:33.099416 20538 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/data/info.pb
I20260619 01:04:33.109799 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20538
I20260619 01:04:33.109997 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal/instance
I20260619 01:04:33.110440 388 external_mini_cluster.cc:1428] Setting key 912bf2b3f70ad6b6ca1e4fb9993e0c5a
I20260619 01:04:33.111649 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.101843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51219 (local address 127.0.97.62:41237)
0619 01:04:33.102071 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.102076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.103441 (+ 1365us) server_negotiation.cc:407] Connection header received
0619 01:04:33.105137 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.105144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.105231 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.105357 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:33.106399 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.107234 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.108803 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.109213 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.109353 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.110740 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:33.110747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:33.110753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:33.110776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:33.110815 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:33.110832 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:33.110955 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.111220 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.111326 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:04:33.112845 20667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.102367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41237 (local address 127.0.97.3:51219)
0619 01:04:33.103288 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.103326 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.104882 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.105593 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.105602 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.105687 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:04:33.106226 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.106242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.107432 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.107446 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.108651 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.108663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.109434 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.109451 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.109668 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.110365 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 01:04:33.110368 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:04:33.110407 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 01:04:33.110488 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:04:33.110959 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:04:33.110970 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:04:33.110979 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.111179 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.111552 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":322,"threads_started":1}
I20260619 01:04:33.114300 20665 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41237
I20260619 01:04:33.114606 20665 heartbeater.cc:464] Registering TS with master...
I20260619 01:04:33.115336 20665 heartbeater.cc:511] Master 127.0.97.62:41237 requested a full tablet report, sending...
I20260619 01:04:33.116649 20218 ts_manager.cc:194] Registered new tserver with Master: 9d6931dcf8954aa3b0bab24adff35b4b (127.0.97.3:38011)
I20260619 01:04:33.117456 20218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:51219
I20260619 01:04:33.125730 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 01:04:33.135298 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.129657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49170 (local address 127.0.97.62:41237)
0619 01:04:33.129843 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.129851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.130171 (+ 320us) server_negotiation.cc:407] Connection header received
0619 01:04:33.130359 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.130366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.130443 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.130545 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:33.131099 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.131796 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.133146 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.133523 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.133648 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.134604 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:33.134608 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:33.134611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:33.134627 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:33.134655 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:33.134666 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:33.134793 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.134928 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.135010 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 01:04:33.138579 20218 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 01:04:33.145081 20218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 01:04:33.169369 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.156685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.0.97.1:38183)
0619 01:04:33.161594 (+ 4909us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.161603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.161632 (+ 29us) server_negotiation.cc:407] Connection header received
0619 01:04:33.161732 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.161740 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.162032 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.162793 (+ 761us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:33.162986 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.163951 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.166743 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.168068 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.168222 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.168501 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.168827 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.168994 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4730,"thread_start_us":5456,"threads_started":1}
I20260619 01:04:33.169422 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.156533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38183 (local address 127.0.0.1:33012)
0619 01:04:33.158515 (+ 1982us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.158532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.158648 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.162273 (+ 3625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.162280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.162309 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:33.162564 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.162573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.164080 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.164087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.166511 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.166527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.168279 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.168286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.168513 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.168634 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.168781 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.168994 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"mutex_wait_us":51,"thread_start_us":290,"threads_started":1}
I20260619 01:04:33.173207 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.158388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45279 (local address 127.0.0.1:46958)
0619 01:04:33.159137 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.159153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.159262 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.165439 (+ 6177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.165444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.165479 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:33.165915 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.165927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.167770 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.167775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.170324 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.170336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.172386 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.172393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.172604 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.172637 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.172731 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.172852 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":143,"thread_start_us":909,"threads_started":1}
I20260619 01:04:33.173334 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.159116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.97.2:45279)
0619 01:04:33.164882 (+ 5766us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.164888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.164910 (+ 22us) server_negotiation.cc:407] Connection header received
0619 01:04:33.164987 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.164991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.165209 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.165561 (+ 352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:33.166086 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.167587 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.170830 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.172194 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.172330 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.172586 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.172790 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.172978 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5615,"thread_start_us":5461,"threads_started":1}
I20260619 01:04:33.173813 20678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.156223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38011 (local address 127.0.0.1:60110)
0619 01:04:33.157321 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.157382 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.157673 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.165134 (+ 7461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.165138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.165160 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:33.166217 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.166228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.167801 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.167807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.170960 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.170972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.173011 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.173017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.173211 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.173242 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.173342 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.173495 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":112,"thread_start_us":1971,"threads_started":1}
I20260619 01:04:33.174144 20339 tablet_service.cc:1467] Processing CreateTablet for tablet dfc19a1256cc4cac895d29097690678d (DEFAULT_TABLE table=test-table [id=79fff78164c84f50a8e08e313e31e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:33.174810 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.156542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.0.97.3:38011)
0619 01:04:33.164555 (+ 8013us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.164566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.164598 (+ 32us) server_negotiation.cc:407] Connection header received
0619 01:04:33.164690 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.164698 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.164926 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.165124 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:33.166413 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.167631 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.171138 (+ 3507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.172500 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.172638 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.172885 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.174034 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.174195 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7839,"thread_start_us":7706,"threads_started":1}
I20260619 01:04:33.175671 20339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc19a1256cc4cac895d29097690678d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:33.176919 20470 tablet_service.cc:1467] Processing CreateTablet for tablet dfc19a1256cc4cac895d29097690678d (DEFAULT_TABLE table=test-table [id=79fff78164c84f50a8e08e313e31e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:33.177591 20600 tablet_service.cc:1467] Processing CreateTablet for tablet dfc19a1256cc4cac895d29097690678d (DEFAULT_TABLE table=test-table [id=79fff78164c84f50a8e08e313e31e983]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 01:04:33.178284 20470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc19a1256cc4cac895d29097690678d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:33.178578 20600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfc19a1256cc4cac895d29097690678d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:33.191283 20683 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Bootstrap starting.
I20260619 01:04:33.192101 20684 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Bootstrap starting.
I20260619 01:04:33.193485 20685 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Bootstrap starting.
I20260619 01:04:33.195264 20684 tablet_bootstrap.cc:654] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:33.196130 20685 tablet_bootstrap.cc:654] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:33.196594 20684 log.cc:824] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:33.197312 20685 log.cc:824] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:33.198217 20683 tablet_bootstrap.cc:654] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:33.199370 20683 log.cc:824] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:33.200186 20684 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: No bootstrap required, opened a new log
I20260619 01:04:33.200299 20685 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: No bootstrap required, opened a new log
I20260619 01:04:33.200488 20684 ts_tablet_manager.cc:1397] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 01:04:33.200577 20685 ts_tablet_manager.cc:1397] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 01:04:33.202301 20683 tablet_bootstrap.cc:492] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: No bootstrap required, opened a new log
I20260619 01:04:33.202553 20683 ts_tablet_manager.cc:1397] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260619 01:04:33.207533 20684 raft_consensus.cc:348] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.208005 20684 raft_consensus.cc:374] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:33.208118 20684 raft_consensus.cc:729] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619bd4c1018442fdaf606163f0e97be3, State: Initialized, Role: FOLLOWER
I20260619 01:04:33.208709 20684 consensus_queue.cc:260] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.209738 20404 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260619 01:04:33.210409 20684 ts_tablet_manager.cc:1428] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 01:04:33.210866 20685 raft_consensus.cc:348] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.211288 20685 raft_consensus.cc:374] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:33.211395 20685 raft_consensus.cc:729] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99234f9760cf4e9084724725e0faf9bf, State: Initialized, Role: FOLLOWER
I20260619 01:04:33.211970 20685 consensus_queue.cc:260] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.212755 20683 raft_consensus.cc:348] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.213001 20535 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260619 01:04:33.213145 20683 raft_consensus.cc:374] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:33.213240 20683 raft_consensus.cc:729] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6931dcf8954aa3b0bab24adff35b4b, State: Initialized, Role: FOLLOWER
I20260619 01:04:33.213626 20685 ts_tablet_manager.cc:1428] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 01:04:33.213763 20683 consensus_queue.cc:260] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.214902 20665 heartbeater.cc:503] Master 127.0.97.62:41237 was elected leader, sending a full tablet report...
I20260619 01:04:33.215508 20683 ts_tablet_manager.cc:1428] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260619 01:04:33.236732 20405 tablet.cc:2369] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 01:04:33.351042 20666 tablet.cc:2369] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:33.368085 20690 raft_consensus.cc:482] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 01:04:33.368397 20690 raft_consensus.cc:504] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.369938 20690 leader_election.cc:288] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d6931dcf8954aa3b0bab24adff35b4b (127.0.97.3:38011), 619bd4c1018442fdaf606163f0e97be3 (127.0.97.1:38183)
I20260619 01:04:33.376888 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.370371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36781 (local address 127.0.97.3:38011)
0619 01:04:33.370573 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.370580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.371545 (+ 965us) server_negotiation.cc:407] Connection header received
0619 01:04:33.371762 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.371769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.371845 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.371956 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:33.372490 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.373182 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.375171 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.375948 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.376099 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.376196 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.376523 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.376647 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:04:33.376897 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.370477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50173 (local address 127.0.97.1:38183)
0619 01:04:33.370693 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.370697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.371581 (+ 884us) server_negotiation.cc:407] Connection header received
0619 01:04:33.371783 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.371807 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.371876 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.371988 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 01:04:33.372633 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.373349 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.375143 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.375904 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.376049 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.376152 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.376520 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.376641 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:04:33.376988 20692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.370443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38011 (local address 127.0.97.2:36781)
0619 01:04:33.371416 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.371442 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.371616 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.371974 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.371978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.372000 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:33.372351 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.372359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.373309 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.373316 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.374964 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.374975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.376184 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.376191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.376362 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.376384 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.376495 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.376615 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":83,"thread_start_us":592,"threads_started":1}
I20260619 01:04:33.377044 20693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.370452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38183 (local address 127.0.97.2:50173)
0619 01:04:33.371474 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 01:04:33.371488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:04:33.371648 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:04:33.372001 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:04:33.372005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:04:33.372026 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 01:04:33.372483 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.372491 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.373506 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.373510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.374961 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:04:33.374973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.376131 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.376137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:04:33.376312 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.376355 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 01:04:33.376482 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 01:04:33.376611 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":64,"spinlock_wait_cycles":22016,"thread_start_us":595,"threads_started":1}
I20260619 01:04:33.377979 20620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc19a1256cc4cac895d29097690678d" candidate_uuid: "99234f9760cf4e9084724725e0faf9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" is_pre_election: true
I20260619 01:04:33.378275 20359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc19a1256cc4cac895d29097690678d" candidate_uuid: "99234f9760cf4e9084724725e0faf9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619bd4c1018442fdaf606163f0e97be3" is_pre_election: true
I20260619 01:04:33.378587 20620 raft_consensus.cc:2454] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99234f9760cf4e9084724725e0faf9bf in term 0.
I20260619 01:04:33.378804 20359 raft_consensus.cc:2454] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99234f9760cf4e9084724725e0faf9bf in term 0.
I20260619 01:04:33.379333 20424 leader_election.cc:302] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99234f9760cf4e9084724725e0faf9bf, 9d6931dcf8954aa3b0bab24adff35b4b; no voters:
I20260619 01:04:33.379740 20690 raft_consensus.cc:2781] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 01:04:33.379856 20690 raft_consensus.cc:482] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 01:04:33.380095 20690 raft_consensus.cc:3037] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:33.382406 20690 raft_consensus.cc:504] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.383116 20690 leader_election.cc:288] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [CANDIDATE]: Term 1 election: Requested vote from peers 9d6931dcf8954aa3b0bab24adff35b4b (127.0.97.3:38011), 619bd4c1018442fdaf606163f0e97be3 (127.0.97.1:38183)
I20260619 01:04:33.383477 20620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc19a1256cc4cac895d29097690678d" candidate_uuid: "99234f9760cf4e9084724725e0faf9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d6931dcf8954aa3b0bab24adff35b4b"
I20260619 01:04:33.383545 20359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfc19a1256cc4cac895d29097690678d" candidate_uuid: "99234f9760cf4e9084724725e0faf9bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619bd4c1018442fdaf606163f0e97be3"
I20260619 01:04:33.383690 20620 raft_consensus.cc:3037] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:33.383718 20359 raft_consensus.cc:3037] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:33.386840 20359 raft_consensus.cc:2454] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99234f9760cf4e9084724725e0faf9bf in term 1.
I20260619 01:04:33.386840 20620 raft_consensus.cc:2454] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99234f9760cf4e9084724725e0faf9bf in term 1.
I20260619 01:04:33.387226 20423 leader_election.cc:302] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 619bd4c1018442fdaf606163f0e97be3, 99234f9760cf4e9084724725e0faf9bf; no voters:
I20260619 01:04:33.387502 20690 raft_consensus.cc:2781] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:33.387889 20690 raft_consensus.cc:686] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 1 LEADER]: Becoming Leader. State: Replica: 99234f9760cf4e9084724725e0faf9bf, State: Running, Role: LEADER
I20260619 01:04:33.388331 20690 consensus_queue.cc:237] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } }
I20260619 01:04:33.393208 20217 catalog_manager.cc:5495] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 99234f9760cf4e9084724725e0faf9bf (127.0.97.2). New cstate: current_term: 1 leader_uuid: "99234f9760cf4e9084724725e0faf9bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99234f9760cf4e9084724725e0faf9bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45279 } health_report { overall_health: HEALTHY } } }
W20260619 01:04:33.417785 20536 tablet.cc:2369] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:04:33.461920 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.456876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.0.97.2:45279)
0619 01:04:33.457061 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.457065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.457084 (+ 19us) server_negotiation.cc:407] Connection header received
0619 01:04:33.457263 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.457267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.457343 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.457544 (+ 201us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:04:33.457829 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.458564 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.460133 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.460536 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.460626 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.461043 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:04:33.461510 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:04:33.461612 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.461684 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.461739 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:04:33.469687 20359 raft_consensus.cc:1261] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 1 FOLLOWER]: Refusing update from remote peer 99234f9760cf4e9084724725e0faf9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:33.469702 20620 raft_consensus.cc:1261] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 1 FOLLOWER]: Refusing update from remote peer 99234f9760cf4e9084724725e0faf9bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:04:33.470665 20694 consensus_queue.cc:1035] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "619bd4c1018442fdaf606163f0e97be3" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:33.471032 20690 consensus_queue.cc:1035] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d6931dcf8954aa3b0bab24adff35b4b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:04:33.479800 20697 mvcc.cc:205] Tried to move back new op lower bound from 7298380076922793984 to 7298380076602740736. Current Snapshot: MvccSnapshot[applied={T|T < 7298380076922793984}]
I20260619 01:04:33.482538 20698 mvcc.cc:205] Tried to move back new op lower bound from 7298380076922793984 to 7298380076602740736. Current Snapshot: MvccSnapshot[applied={T|T < 7298380076922793984}]
I20260619 01:04:33.483712 20699 mvcc.cc:205] Tried to move back new op lower bound from 7298380076922793984 to 7298380076602740736. Current Snapshot: MvccSnapshot[applied={T|T < 7298380076922793984}]
I20260619 01:04:33.493726 20217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:33.493974 20217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 01:04:33.497339 20217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5b5c4c337924ed587233408c527be5f: Sending DeleteTablet for 3 replicas of tablet dfc19a1256cc4cac895d29097690678d
I20260619 01:04:33.498401 20339 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc19a1256cc4cac895d29097690678d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:33 UTC) from {username='slave'} at 127.0.0.1:33012
I20260619 01:04:33.498399 20600 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc19a1256cc4cac895d29097690678d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:33 UTC) from {username='slave'} at 127.0.0.1:60110
I20260619 01:04:33.498481 20470 tablet_service.cc:1514] Processing DeleteTablet for tablet dfc19a1256cc4cac895d29097690678d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 01:04:33 UTC) from {username='slave'} at 127.0.0.1:46958
I20260619 01:04:33.500447 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20277
I20260619 01:04:33.511510 20708 tablet_replica.cc:330] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: stopping tablet replica
I20260619 01:04:33.511729 20706 tablet_replica.cc:330] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: stopping tablet replica
I20260619 01:04:33.512442 20706 raft_consensus.cc:2229] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:33.513130 20706 raft_consensus.cc:2258] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:33.515528 20706 ts_tablet_manager.cc:1905] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:33.512579 20708 raft_consensus.cc:2229] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 1 LEADER]: Raft consensus shutting down.
I20260619 01:04:33.518478 20708 raft_consensus.cc:2258] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:33.520670 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.501811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33016 (local address 127.0.97.1:38183)
0619 01:04:33.502002 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.502008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.507266 (+ 5258us) server_negotiation.cc:407] Connection header received
0619 01:04:33.507476 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.507482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.507605 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.507715 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:33.513300 (+ 5585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.514417 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.516246 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.516953 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.517054 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.518675 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:33.518685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:33.518700 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:33.519497 (+ 797us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:33.519554 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:33.519581 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:33.519679 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.520167 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.520282 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:04:33.521019 20707 tablet_replica.cc:330] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: stopping tablet replica
I20260619 01:04:33.521364 20399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:33.521453 20707 raft_consensus.cc:2229] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 01:04:33.521667 20708 ts_tablet_manager.cc:1905] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:33.522027 20707 raft_consensus.cc:2258] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 01:04:33.524999 20707 ts_tablet_manager.cc:1905] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 01:04:33.529590 20708 ts_tablet_manager.cc:1918] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:33.529784 20708 log.cc:1196] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-1/wal/wals/dfc19a1256cc4cac895d29097690678d
I20260619 01:04:33.530305 20708 ts_tablet_manager.cc:1939] T dfc19a1256cc4cac895d29097690678d P 99234f9760cf4e9084724725e0faf9bf: Deleting consensus metadata
I20260619 01:04:33.531850 20707 ts_tablet_manager.cc:1918] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:33.532017 20707 log.cc:1196] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-2/wal/wals/dfc19a1256cc4cac895d29097690678d
I20260619 01:04:33.532174 20205 catalog_manager.cc:4841] TS 99234f9760cf4e9084724725e0faf9bf (127.0.97.2:45279): tablet dfc19a1256cc4cac895d29097690678d (table test-table [id=79fff78164c84f50a8e08e313e31e983]) successfully deleted
I20260619 01:04:33.532526 20707 ts_tablet_manager.cc:1939] T dfc19a1256cc4cac895d29097690678d P 9d6931dcf8954aa3b0bab24adff35b4b: Deleting consensus metadata
I20260619 01:04:33.534261 20203 catalog_manager.cc:4841] TS 9d6931dcf8954aa3b0bab24adff35b4b (127.0.97.3:38011): tablet dfc19a1256cc4cac895d29097690678d (table test-table [id=79fff78164c84f50a8e08e313e31e983]) successfully deleted
I20260619 01:04:33.617449 20706 ts_tablet_manager.cc:1918] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 01:04:33.617656 20706 log.cc:1196] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Deleting WAL directory at /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781830760188020-388-0/minicluster-data/ts-0/wal/wals/dfc19a1256cc4cac895d29097690678d
I20260619 01:04:33.618206 20706 ts_tablet_manager.cc:1939] T dfc19a1256cc4cac895d29097690678d P 619bd4c1018442fdaf606163f0e97be3: Deleting consensus metadata
I20260619 01:04:33.619931 20202 catalog_manager.cc:4841] TS 619bd4c1018442fdaf606163f0e97be3 (127.0.97.1:38183): tablet dfc19a1256cc4cac895d29097690678d (table test-table [id=79fff78164c84f50a8e08e313e31e983]) successfully deleted
I20260619 01:04:33.619998 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20277
I20260619 01:04:33.637168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20408
I20260619 01:04:33.644613 20682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.637904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.0.97.2:45279)
0619 01:04:33.638093 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.638099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.638119 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:04:33.638282 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.638288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.638384 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.638495 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:33.638912 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.640009 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.641180 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.641762 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.641876 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.643077 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:33.643090 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:33.643108 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:33.643900 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:33.643967 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:33.643992 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:33.644104 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.644186 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.644270 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:04:33.645305 20530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:33.756521 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20408
I20260619 01:04:33.774540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20538
I20260619 01:04:33.781320 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:33.775286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.0.97.3:38011)
0619 01:04:33.775476 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:33.775483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:33.775568 (+ 85us) server_negotiation.cc:407] Connection header received
0619 01:04:33.775785 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:33.775789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:33.775869 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:33.775980 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:33.776391 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.777098 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.778401 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:33.778837 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:33.778968 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:33.780063 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:33.780073 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:33.780086 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:33.780634 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:33.780681 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:33.780699 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:33.780818 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:33.780915 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 01:04:33.781024 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:04:33.782037 20660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:33.868664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20538
I20260619 01:04:33.885095 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20187
I20260619 01:04:33.886108 20248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:33.993203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20187
2026-06-19T01:04:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2755 ms)
[----------] 6 tests from EncryptionPolicyTest (20928 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:34 dist-test-slave-3rxd krb5kdc[20716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:34 dist-test-slave-3rxd krb5kdc[20716](info): set up 2 sockets
Jun 19 01:04:34 dist-test-slave-3rxd krb5kdc[20716](info): commencing operation
krb5kdc: starting...
W20260619 01:04:35.091519 388 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:35 dist-test-slave-3rxd krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:35.326573 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:35.648829 20729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:35.649155 20729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:35.649333 20729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:35.656442 20729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:35.656605 20729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:35.656714 20729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:35.656814 20729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:35.656883 20729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:35.666388 20729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:35.667701 20729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:35.668970 20729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:35.680828 20734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:35.680936 20735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:35.680876 20737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:35.684103 20729 server_base.cc:1034] running on GCE node
I20260619 01:04:35.684816 20729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:35.686122 20729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:35.687323 20729 hybrid_clock.cc:648] HybridClock initialized: now 1781831075687265 us; error 41 us; skew 500 ppm
I20260619 01:04:35.687678 20729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:35 dist-test-slave-3rxd krb5kdc[20716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:35.695228 20729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:35.696805 20729 webserver.cc:460] Webserver started at http://127.0.0.1:33593/ using document root <none> and password file <none>
I20260619 01:04:35.697537 20729 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:35.697667 20729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:35.697993 20729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:35.700989 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "764f1e9bc3314893bc0e075ee439f4ee"
format_stamp: "Formatted at 2026-06-19 01:04:35 on dist-test-slave-3rxd"
server_key: "e1518200d0e28467b41487f94bf01bc6"
server_key_iv: "2ea5945119e3c0f5f5b6dc908f205488"
server_key_version: "encryptionkey@0"
I20260619 01:04:35.701746 20729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "764f1e9bc3314893bc0e075ee439f4ee"
format_stamp: "Formatted at 2026-06-19 01:04:35 on dist-test-slave-3rxd"
server_key: "e1518200d0e28467b41487f94bf01bc6"
server_key_iv: "2ea5945119e3c0f5f5b6dc908f205488"
server_key_version: "encryptionkey@0"
I20260619 01:04:35.706894 20729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:04:35.710551 20744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:35.712080 20729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:04:35.712270 20729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "764f1e9bc3314893bc0e075ee439f4ee"
format_stamp: "Formatted at 2026-06-19 01:04:35 on dist-test-slave-3rxd"
server_key: "e1518200d0e28467b41487f94bf01bc6"
server_key_iv: "2ea5945119e3c0f5f5b6dc908f205488"
server_key_version: "encryptionkey@0"
I20260619 01:04:35.712463 20729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:35.737943 20729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:35.741815 20729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:35.742110 20729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:35.760174 20729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36635
I20260619 01:04:35.760214 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36635 every 8 connection(s)
I20260619 01:04:35.761451 20729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:35.764827 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20729
I20260619 01:04:35.765046 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:35.765828 20796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:35.765867 388 external_mini_cluster.cc:1428] Setting key cb7ba82afac8ae4d9e3eadd361da31ec
I20260619 01:04:35.779906 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee: Bootstrap starting.
Jun 19 01:04:35 dist-test-slave-3rxd krb5kdc[20716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:04:35.783020 20796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:35.784063 20796 log.cc:824] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:35.786682 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee: No bootstrap required, opened a new log
I20260619 01:04:35.788825 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:35.767738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40657 (local address 127.0.0.1:36635)
0619 01:04:35.768536 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:35.768549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:35.768586 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:04:35.770343 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:35.770365 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:35.770802 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:35.771128 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:35.772817 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:35.774051 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:35.775838 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:35.776487 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:35.776609 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:35.781305 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:35.781311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:35.781323 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:35.781354 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:35.784432 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:35.785215 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:35.785220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:35.785226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:35.785383 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:35.785861 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:35.785864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:35.785866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:35.786315 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:35.786634 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:35.787219 (+ 585us) server_negotiation.cc:299] Negotiation successful
0619 01:04:35.787546 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":368,"threads_started":1}
I20260619 01:04:35.792647 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:35.796204 20796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260619 01:04:35.796701 20796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:35.796813 20796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764f1e9bc3314893bc0e075ee439f4ee, State: Initialized, Role: FOLLOWER
I20260619 01:04:35.797487 20796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260619 01:04:35.797765 20796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:35.797914 20796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:35.798130 20796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:35.801451 20796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260619 01:04:35.801975 20796 leader_election.cc:302] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764f1e9bc3314893bc0e075ee439f4ee; no voters:
I20260619 01:04:35.802594 20796 leader_election.cc:288] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:35.802670 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:35.804167 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [term 1 LEADER]: Becoming Leader. State: Replica: 764f1e9bc3314893bc0e075ee439f4ee, State: Running, Role: LEADER
I20260619 01:04:35.804777 20796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:35.804708 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260619 01:04:35.813598 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764f1e9bc3314893bc0e075ee439f4ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260619 01:04:35.813985 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:35.814376 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764f1e9bc3314893bc0e075ee439f4ee. Latest consensus state: current_term: 1 leader_uuid: "764f1e9bc3314893bc0e075ee439f4ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764f1e9bc3314893bc0e075ee439f4ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260619 01:04:35.814666 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:35.817770 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:35.798368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44085 (local address 127.0.0.1:36635)
0619 01:04:35.798575 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:35.798583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:35.798957 (+ 374us) server_negotiation.cc:407] Connection header received
0619 01:04:35.799114 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:35.799117 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:35.799175 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:35.799254 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:35.801054 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:35.801813 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:35.803685 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:35.804376 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:35.804479 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:35.808113 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:35.808122 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:35.808129 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:35.808177 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:35.812915 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:35.813834 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:35.813842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:35.813853 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:35.813993 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:35.814606 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:35.814613 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:35.814618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:35.815066 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:35.815460 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:35.817004 (+ 1544us) server_negotiation.cc:299] Negotiation successful
0619 01:04:35.817234 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":57}
I20260619 01:04:35.820150 20816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:35.824558 20816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:35.833155 20816 catalog_manager.cc:1347] Generated new cluster ID: b02eb704c9cf4414bdc1da96520b3e2f
I20260619 01:04:35.833261 20816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:35.844942 20816 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:35.846091 20816 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:35.856591 20816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 764f1e9bc3314893bc0e075ee439f4ee: Generated new TSK 0
I20260619 01:04:35.857303 20816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:35.866660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20729
I20260619 01:04:35.868198 20791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:35.956452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20729
W20260619 01:04:35.970209 20714 connection.cc:535] client connection to 127.0.0.1:36635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36635): Connection reset by peer (error 104)
2026-06-19T01:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:36 dist-test-slave-3rxd krb5kdc[20828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:36 dist-test-slave-3rxd krb5kdc[20828](info): set up 2 sockets
Jun 19 01:04:36 dist-test-slave-3rxd krb5kdc[20828](info): commencing operation
krb5kdc: starting...
W20260619 01:04:37.019168 388 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:37 dist-test-slave-3rxd krb5kdc[20828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:37.246982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:37.560547 20841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:37.560820 20841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:37.560945 20841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:37.567795 20841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:37.567924 20841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:37.568024 20841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:37.568115 20841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:37.568190 20841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:37.577236 20841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:37.578403 20841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:37.579676 20841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:37.591084 20849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:37.591190 20846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:37.591171 20847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:37.591903 20841 server_base.cc:1034] running on GCE node
I20260619 01:04:37.592607 20841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:37.593868 20841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:37.595090 20841 hybrid_clock.cc:648] HybridClock initialized: now 1781831077595059 us; error 51 us; skew 500 ppm
I20260619 01:04:37.595448 20841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:37 dist-test-slave-3rxd krb5kdc[20828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:37.598364 20841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:37.599818 20841 webserver.cc:460] Webserver started at http://127.0.0.1:41223/ using document root <none> and password file <none>
I20260619 01:04:37.600446 20841 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:37.600607 20841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:37.600903 20841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:37.603645 20841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "662e903cac104245ae99160cd525aa94"
format_stamp: "Formatted at 2026-06-19 01:04:37 on dist-test-slave-3rxd"
server_key: "8e57c51375915f5955cdb5543279c0d4"
server_key_iv: "39f6e19a37db928ae2d7fedb5e53101e"
server_key_version: "encryptionkey@0"
I20260619 01:04:37.604346 20841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "662e903cac104245ae99160cd525aa94"
format_stamp: "Formatted at 2026-06-19 01:04:37 on dist-test-slave-3rxd"
server_key: "8e57c51375915f5955cdb5543279c0d4"
server_key_iv: "39f6e19a37db928ae2d7fedb5e53101e"
server_key_version: "encryptionkey@0"
I20260619 01:04:37.609035 20841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:37.612239 20856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:37.613487 20841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:04:37.613662 20841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "662e903cac104245ae99160cd525aa94"
format_stamp: "Formatted at 2026-06-19 01:04:37 on dist-test-slave-3rxd"
server_key: "8e57c51375915f5955cdb5543279c0d4"
server_key_iv: "39f6e19a37db928ae2d7fedb5e53101e"
server_key_version: "encryptionkey@0"
I20260619 01:04:37.613852 20841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:37.630985 20841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:37.634902 20841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:37.635186 20841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:37.652760 20841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35877
I20260619 01:04:37.652791 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35877 every 8 connection(s)
I20260619 01:04:37.654130 20841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:37.657351 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20841
I20260619 01:04:37.657539 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:37.657896 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:37.658097 388 external_mini_cluster.cc:1428] Setting key a47def395fbb75737fe79f7e1853eafe
I20260619 01:04:37.669821 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94: Bootstrap starting.
I20260619 01:04:37.670899 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:37.659727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52515 (local address 127.0.0.1:35877)
0619 01:04:37.660426 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:37.660438 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:37.660476 (+ 38us) server_negotiation.cc:407] Connection header received
0619 01:04:37.662461 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:37.662480 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:37.662905 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:37.663294 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:37.663801 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:37.665073 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:37.666384 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:37.667146 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:37.667351 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:37.668615 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:37.668622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:37.668634 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:37.669184 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:37.669273 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:37.669303 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:37.669510 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:37.669763 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 01:04:37.669945 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":316,"threads_started":1}
I20260619 01:04:37.672878 20908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:37.673965 20908 log.cc:824] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:37.674358 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:37.676463 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94: No bootstrap required, opened a new log
I20260619 01:04:37.685262 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:37.678432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44011 (local address 127.0.0.1:35877)
0619 01:04:37.678650 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:37.678658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:37.679111 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:04:37.679254 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:37.679257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:37.679311 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:37.679386 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:37.679991 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:37.680706 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:37.682572 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:37.683093 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:37.683212 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:37.684449 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:37.684455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:37.684458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:37.684475 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:37.684530 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:37.684546 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:37.684642 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:37.684883 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 01:04:37.684970 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:04:37.685753 20908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260619 01:04:37.686120 20908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:37.686245 20908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662e903cac104245ae99160cd525aa94, State: Initialized, Role: FOLLOWER
I20260619 01:04:37.686892 20908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260619 01:04:37.687145 20908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:37.687281 20908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:37.687446 20908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:37.689477 20908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260619 01:04:37.689937 20908 leader_election.cc:302] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662e903cac104245ae99160cd525aa94; no voters:
I20260619 01:04:37.690478 20908 leader_election.cc:288] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:37.690584 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:37.691284 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [term 1 LEADER]: Becoming Leader. State: Replica: 662e903cac104245ae99160cd525aa94, State: Running, Role: LEADER
I20260619 01:04:37.691912 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } }
I20260619 01:04:37.692595 20908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:37.694933 20919 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "662e903cac104245ae99160cd525aa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260619 01:04:37.695500 20919 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:37.695067 20920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662e903cac104245ae99160cd525aa94. Latest consensus state: current_term: 1 leader_uuid: "662e903cac104245ae99160cd525aa94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662e903cac104245ae99160cd525aa94" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35877 } } }
I20260619 01:04:37.695757 20920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:37.696935 20927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:37.702133 20927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:37.710240 20927 catalog_manager.cc:1347] Generated new cluster ID: 2933a7a9d0d1428185b6ca44a7299aed
I20260619 01:04:37.710337 20927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:37.722167 20927 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:37.723524 20927 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:37.732776 20927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 662e903cac104245ae99160cd525aa94: Generated new TSK 0
I20260619 01:04:37.733560 20927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:37.746202 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20841
W20260619 01:04:37.747321 20903 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52515
W20260619 01:04:37.748041 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20841: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:37.748219 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20841
W20260619 01:04:37.765434 20826 connection.cc:535] client connection to 127.0.0.1:35877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35877): Connection reset by peer (error 104)
2026-06-19T01:04:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1795 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:37 dist-test-slave-3rxd krb5kdc[20939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:37 dist-test-slave-3rxd krb5kdc[20939](info): set up 2 sockets
Jun 19 01:04:37 dist-test-slave-3rxd krb5kdc[20939](info): commencing operation
krb5kdc: starting...
W20260619 01:04:38.879019 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:39 dist-test-slave-3rxd krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:39.115831 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35899
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:39.446353 20952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:39.446734 20952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:39.446928 20952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:39.456888 20952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:39.457034 20952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:39.457113 20952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:39.457198 20952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:39.457263 20952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:39.468089 20952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:39.469334 20952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:39.470676 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:39.482050 20957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:39.482137 20960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:39.482177 20958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:39.483052 20952 server_base.cc:1034] running on GCE node
I20260619 01:04:39.483700 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:39.484951 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:39.486158 20952 hybrid_clock.cc:648] HybridClock initialized: now 1781831079486117 us; error 36 us; skew 500 ppm
I20260619 01:04:39.486505 20952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:39 dist-test-slave-3rxd krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831079, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:39.489566 20952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:39.491026 20952 webserver.cc:460] Webserver started at http://127.0.0.1:42897/ using document root <none> and password file <none>
I20260619 01:04:39.491659 20952 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:39.491799 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:39.492102 20952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:39.494932 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "0354b193c33e4460b543c8ebaf5f4f96"
format_stamp: "Formatted at 2026-06-19 01:04:39 on dist-test-slave-3rxd"
server_key: "00a4e3e1f206114d7175abe8a6b61700"
server_key_iv: "531918dc8383965360b5146e0c53ef7e"
server_key_version: "encryptionkey@0"
I20260619 01:04:39.495652 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "0354b193c33e4460b543c8ebaf5f4f96"
format_stamp: "Formatted at 2026-06-19 01:04:39 on dist-test-slave-3rxd"
server_key: "00a4e3e1f206114d7175abe8a6b61700"
server_key_iv: "531918dc8383965360b5146e0c53ef7e"
server_key_version: "encryptionkey@0"
I20260619 01:04:39.500347 20952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 01:04:39.503582 20967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:39.504770 20952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:04:39.504954 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "0354b193c33e4460b543c8ebaf5f4f96"
format_stamp: "Formatted at 2026-06-19 01:04:39 on dist-test-slave-3rxd"
server_key: "00a4e3e1f206114d7175abe8a6b61700"
server_key_iv: "531918dc8383965360b5146e0c53ef7e"
server_key_version: "encryptionkey@0"
I20260619 01:04:39.505149 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:39.511869 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:39.517688 20952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:39.517987 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:39.534829 20952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35899
I20260619 01:04:39.534874 21018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35899 every 8 connection(s)
I20260619 01:04:39.536173 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:39.536708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 20952
I20260619 01:04:39.536952 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:39.537493 388 external_mini_cluster.cc:1428] Setting key 2a8ec9cbd82c3b675b5f81c28c9c3d2a
I20260619 01:04:39.540666 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:39.545737 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:39.539244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51181 (local address 127.0.0.1:35899)
0619 01:04:39.539955 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:39.539969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:39.540013 (+ 44us) server_negotiation.cc:407] Connection header received
0619 01:04:39.542234 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:39.542258 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:39.542643 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:39.543058 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:39.543960 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:39.543969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:39.543984 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:39.544000 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:39.544091 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:39.544125 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:39.544313 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:39.544429 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 01:04:39.544615 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":281,"threads_started":1}
I20260619 01:04:39.549125 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:39.553458 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96: Bootstrap starting.
I20260619 01:04:39.556577 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:39.554213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45313 (local address 127.0.0.1:35899)
0619 01:04:39.554368 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:39.554374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:39.554779 (+ 405us) server_negotiation.cc:407] Connection header received
0619 01:04:39.554961 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:39.554966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:39.555025 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:39.555129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:39.555948 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:39.555954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:39.555959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:39.555975 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:39.556024 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:39.556046 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:39.556153 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:39.556246 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 01:04:39.556296 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 01:04:39.556908 21019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:39.557886 21019 log.cc:824] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:39.560310 21019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96: No bootstrap required, opened a new log
I20260619 01:04:39.567195 21019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } }
I20260619 01:04:39.567533 21019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:39.567651 21019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0354b193c33e4460b543c8ebaf5f4f96, State: Initialized, Role: FOLLOWER
I20260619 01:04:39.568194 21019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } }
I20260619 01:04:39.568413 21019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:39.568611 21019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:39.568789 21019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:39.570925 21019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } }
I20260619 01:04:39.571393 21019 leader_election.cc:302] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0354b193c33e4460b543c8ebaf5f4f96; no voters:
I20260619 01:04:39.571966 21019 leader_election.cc:288] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:39.572063 21029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:39.572763 21029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [term 1 LEADER]: Becoming Leader. State: Replica: 0354b193c33e4460b543c8ebaf5f4f96, State: Running, Role: LEADER
I20260619 01:04:39.573343 21029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } }
I20260619 01:04:39.574007 21019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:39.576144 21031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0354b193c33e4460b543c8ebaf5f4f96. Latest consensus state: current_term: 1 leader_uuid: "0354b193c33e4460b543c8ebaf5f4f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } } }
I20260619 01:04:39.576398 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0354b193c33e4460b543c8ebaf5f4f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354b193c33e4460b543c8ebaf5f4f96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35899 } } }
I20260619 01:04:39.576700 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:39.576674 21031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:39.578382 21035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:39.583464 21035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:39.592298 21035 catalog_manager.cc:1347] Generated new cluster ID: 8748bec44eda4145a4622bc824199b01
I20260619 01:04:39.592417 21035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:39.615463 21035 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:39.615912 21035 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:39.624452 21035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0354b193c33e4460b543c8ebaf5f4f96: Generated new TSK 0
I20260619 01:04:39.625406 21035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:39.640513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20952
W20260619 01:04:39.641497 21014 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51181
W20260619 01:04:39.642165 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 20952: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:39.642298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 20952
2026-06-19T01:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:39 dist-test-slave-3rxd krb5kdc[21050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:39 dist-test-slave-3rxd krb5kdc[21050](info): set up 2 sockets
Jun 19 01:04:39 dist-test-slave-3rxd krb5kdc[21050](info): commencing operation
krb5kdc: starting...
W20260619 01:04:40.735323 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:40 dist-test-slave-3rxd krb5kdc[21050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:40.979975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:41.299455 21063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:41.299768 21063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:41.299932 21063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:41.306897 21063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:41.307030 21063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:41.307132 21063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:41.307235 21063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:41.307317 21063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:41.317090 21063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:41.318439 21063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:41.319782 21063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:41.331761 21069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:41.331791 21068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:41.331768 21071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:41.332690 21063 server_base.cc:1034] running on GCE node
I20260619 01:04:41.333546 21063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:41.335094 21063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:41.336323 21063 hybrid_clock.cc:648] HybridClock initialized: now 1781831081336237 us; error 79 us; skew 500 ppm
I20260619 01:04:41.336681 21063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:41 dist-test-slave-3rxd krb5kdc[21050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:41.342880 21063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:41.344336 21063 webserver.cc:460] Webserver started at http://127.0.0.1:40071/ using document root <none> and password file <none>
I20260619 01:04:41.345038 21063 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:41.345178 21063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:41.345484 21063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:41.348552 21063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "e639accb9449447ebc002a0136489376"
format_stamp: "Formatted at 2026-06-19 01:04:41 on dist-test-slave-3rxd"
server_key: "0deeffd116dbf116570ac8e37c6bd2e3"
server_key_iv: "bfc9f5baee665fa9e523c2c5188df242"
server_key_version: "encryptionkey@0"
I20260619 01:04:41.349303 21063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "e639accb9449447ebc002a0136489376"
format_stamp: "Formatted at 2026-06-19 01:04:41 on dist-test-slave-3rxd"
server_key: "0deeffd116dbf116570ac8e37c6bd2e3"
server_key_iv: "bfc9f5baee665fa9e523c2c5188df242"
server_key_version: "encryptionkey@0"
I20260619 01:04:41.354521 21063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 01:04:41.357923 21078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:41.359288 21063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:04:41.359476 21063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "e639accb9449447ebc002a0136489376"
format_stamp: "Formatted at 2026-06-19 01:04:41 on dist-test-slave-3rxd"
server_key: "0deeffd116dbf116570ac8e37c6bd2e3"
server_key_iv: "bfc9f5baee665fa9e523c2c5188df242"
server_key_version: "encryptionkey@0"
I20260619 01:04:41.359670 21063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:41.375735 21063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:41.380266 21063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:41.380563 21063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:41.399176 21063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33259
I20260619 01:04:41.399199 21129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33259 every 8 connection(s)
I20260619 01:04:41.400543 21063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:41.403517 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21063
I20260619 01:04:41.403700 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:41.404250 388 external_mini_cluster.cc:1428] Setting key 27c4d5fb3cf1db3c7d20e2c95641f8c9
I20260619 01:04:41.404678 21130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:41.418967 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376: Bootstrap starting.
Jun 19 01:04:41 dist-test-slave-3rxd krb5kdc[21050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:04:41.422070 21130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:41.423178 21130 log.cc:824] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:41.425925 21130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376: No bootstrap required, opened a new log
I20260619 01:04:41.427457 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:41.406004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36257 (local address 127.0.0.1:33259)
0619 01:04:41.406805 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:41.406816 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:41.406856 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:04:41.409177 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:41.409198 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:41.409603 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:41.409911 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:41.411732 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:41.412959 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:41.414807 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:41.415482 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:41.415613 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:41.420285 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:41.420296 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:41.420309 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:41.420345 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:41.423534 (+ 3189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:41.424228 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:41.424234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:41.424240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:41.424399 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:41.424793 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:41.424798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:41.424800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:41.425294 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:41.425598 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:41.426070 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 01:04:41.426464 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":372,"threads_started":1}
I20260619 01:04:41.430962 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:41.433892 21130 raft_consensus.cc:348] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260619 01:04:41.434242 21130 raft_consensus.cc:374] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:41.434360 21130 raft_consensus.cc:729] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e639accb9449447ebc002a0136489376, State: Initialized, Role: FOLLOWER
I20260619 01:04:41.434976 21130 consensus_queue.cc:260] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260619 01:04:41.435307 21130 raft_consensus.cc:388] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:41.435472 21130 raft_consensus.cc:482] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:41.435660 21130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:41.438660 21130 raft_consensus.cc:504] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260619 01:04:41.439277 21130 leader_election.cc:302] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e639accb9449447ebc002a0136489376; no voters:
I20260619 01:04:41.439899 21130 leader_election.cc:288] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:41.440032 21140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:41.440644 21140 raft_consensus.cc:686] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [term 1 LEADER]: Becoming Leader. State: Replica: e639accb9449447ebc002a0136489376, State: Running, Role: LEADER
I20260619 01:04:41.441200 21140 consensus_queue.cc:237] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } }
I20260619 01:04:41.443270 21141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e639accb9449447ebc002a0136489376. Latest consensus state: current_term: 1 leader_uuid: "e639accb9449447ebc002a0136489376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260619 01:04:41.443769 21141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:41.443905 21142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e639accb9449447ebc002a0136489376" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e639accb9449447ebc002a0136489376" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33259 } } }
I20260619 01:04:41.444201 21142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:41.446972 21130 sys_catalog.cc:564] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:41.447153 21144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:41.453766 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:41.435242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47051 (local address 127.0.0.1:33259)
0619 01:04:41.435427 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:41.435435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:41.435928 (+ 493us) server_negotiation.cc:407] Connection header received
0619 01:04:41.436106 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:41.436109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:41.436167 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:41.436242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:41.437957 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:41.438618 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:41.440552 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:41.441538 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:41.441666 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:41.445376 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:41.445385 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:41.445392 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:41.445440 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:41.449465 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:41.450560 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:41.450569 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:41.450575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:41.450781 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:41.451367 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:41.451372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:41.451375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:41.451788 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:41.452054 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:41.453228 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0619 01:04:41.453441 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:04:41.454780 21144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:41.463317 21144 catalog_manager.cc:1347] Generated new cluster ID: 72bf97745d8e40b89417153b19ad0c51
I20260619 01:04:41.463510 21144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:41.499047 21144 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:41.500320 21144 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:41.517917 21144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e639accb9449447ebc002a0136489376: Generated new TSK 0
I20260619 01:04:41.519014 21144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:41.532457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21063
I20260619 01:04:41.533524 21125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:41.626400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21063
2026-06-19T01:04:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1981 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:41 dist-test-slave-3rxd krb5kdc[21162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:41 dist-test-slave-3rxd krb5kdc[21162](info): set up 2 sockets
Jun 19 01:04:41 dist-test-slave-3rxd krb5kdc[21162](info): commencing operation
krb5kdc: starting...
W20260619 01:04:42.731088 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:42 dist-test-slave-3rxd krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:42.959986 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:43.275514 21175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:43.275843 21175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:43.276005 21175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:43.282850 21175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:43.282999 21175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:43.283092 21175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:43.283200 21175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:43.283284 21175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:43.292742 21175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:43.293916 21175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:43.295219 21175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:43.306692 21180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:43.306654 21181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:43.306994 21183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:43.307466 21175 server_base.cc:1034] running on GCE node
I20260619 01:04:43.308176 21175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:43.309407 21175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:43.310609 21175 hybrid_clock.cc:648] HybridClock initialized: now 1781831083310563 us; error 31 us; skew 500 ppm
I20260619 01:04:43.310983 21175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:43 dist-test-slave-3rxd krb5kdc[21162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:43.313908 21175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:43.315405 21175 webserver.cc:460] Webserver started at http://127.0.0.1:43951/ using document root <none> and password file <none>
I20260619 01:04:43.316057 21175 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:43.316197 21175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:43.316498 21175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:43.319402 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "dfcf66f3188542068c56dc37ada78afe"
format_stamp: "Formatted at 2026-06-19 01:04:43 on dist-test-slave-3rxd"
server_key: "d99e714090bdb9142696fb1b13bd52d2"
server_key_iv: "4858c270711bdebe0b6372befd99d19b"
server_key_version: "encryptionkey@0"
I20260619 01:04:43.320156 21175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "dfcf66f3188542068c56dc37ada78afe"
format_stamp: "Formatted at 2026-06-19 01:04:43 on dist-test-slave-3rxd"
server_key: "d99e714090bdb9142696fb1b13bd52d2"
server_key_iv: "4858c270711bdebe0b6372befd99d19b"
server_key_version: "encryptionkey@0"
I20260619 01:04:43.325018 21175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:43.328328 21190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:43.329504 21175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:04:43.329677 21175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "dfcf66f3188542068c56dc37ada78afe"
format_stamp: "Formatted at 2026-06-19 01:04:43 on dist-test-slave-3rxd"
server_key: "d99e714090bdb9142696fb1b13bd52d2"
server_key_iv: "4858c270711bdebe0b6372befd99d19b"
server_key_version: "encryptionkey@0"
I20260619 01:04:43.329860 21175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:43.346236 21175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:43.349787 21175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:43.350052 21175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:43.367308 21175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32973
I20260619 01:04:43.367352 21241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32973 every 8 connection(s)
I20260619 01:04:43.368573 21175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:43.371665 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21175
I20260619 01:04:43.371848 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:43.372331 21242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:43.372423 388 external_mini_cluster.cc:1428] Setting key f3b45b6aba97933e0cbcd131399778f8
I20260619 01:04:43.385005 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:43.374013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58131 (local address 127.0.0.1:32973)
0619 01:04:43.374787 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:43.374804 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:43.374841 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:04:43.376795 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:43.376815 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:43.377189 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:43.377497 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:43.378023 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:43.379372 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:43.381213 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:43.381768 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:43.381896 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:43.383016 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:43.383026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:43.383037 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:43.383535 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:43.383604 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:43.383626 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:43.383714 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:43.383926 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 01:04:43.384150 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260619 01:04:43.385532 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe: Bootstrap starting.
I20260619 01:04:43.388545 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:43.388906 21242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:43.389840 21242 log.cc:824] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:43.392656 21242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe: No bootstrap required, opened a new log
I20260619 01:04:43.399598 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:43.392967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57827 (local address 127.0.0.1:32973)
0619 01:04:43.393179 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:43.393190 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:43.393570 (+ 380us) server_negotiation.cc:407] Connection header received
0619 01:04:43.393760 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:43.393764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:43.393816 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:43.393890 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:43.394499 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:43.395286 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:43.397182 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:43.397643 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:43.397738 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:43.398903 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:43.398907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:43.398910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:43.398926 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:43.398963 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:43.398977 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:43.399051 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:43.399201 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 01:04:43.399266 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 01:04:43.401966 21242 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260619 01:04:43.402287 21242 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:43.402420 21242 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcf66f3188542068c56dc37ada78afe, State: Initialized, Role: FOLLOWER
I20260619 01:04:43.403007 21242 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260619 01:04:43.403235 21242 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:43.403380 21242 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:43.403553 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:43.405602 21242 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260619 01:04:43.406090 21242 leader_election.cc:302] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcf66f3188542068c56dc37ada78afe; no voters:
I20260619 01:04:43.406610 21242 leader_election.cc:288] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:43.406698 21252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:43.407435 21252 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [term 1 LEADER]: Becoming Leader. State: Replica: dfcf66f3188542068c56dc37ada78afe, State: Running, Role: LEADER
I20260619 01:04:43.408015 21252 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } }
I20260619 01:04:43.408672 21242 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:43.410764 21254 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcf66f3188542068c56dc37ada78afe. Latest consensus state: current_term: 1 leader_uuid: "dfcf66f3188542068c56dc37ada78afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } } }
I20260619 01:04:43.411270 21254 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:43.410818 21253 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcf66f3188542068c56dc37ada78afe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcf66f3188542068c56dc37ada78afe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32973 } } }
I20260619 01:04:43.411928 21253 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:43.412943 21260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:43.417655 21260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:43.425828 21260 catalog_manager.cc:1347] Generated new cluster ID: 6a4370d6deed472f84295b9c0d3b8ae4
I20260619 01:04:43.425926 21260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:43.435858 21260 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:43.437378 21260 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:43.443441 21260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcf66f3188542068c56dc37ada78afe: Generated new TSK 0
I20260619 01:04:43.444146 21260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:43.457806 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21175
W20260619 01:04:43.458873 21237 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58131
W20260619 01:04:43.459558 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21175: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:43.459702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21175
2026-06-19T01:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1836 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:43 dist-test-slave-3rxd krb5kdc[21273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:43 dist-test-slave-3rxd krb5kdc[21273](info): set up 2 sockets
Jun 19 01:04:43 dist-test-slave-3rxd krb5kdc[21273](info): commencing operation
krb5kdc: starting...
W20260619 01:04:44.543129 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:44 dist-test-slave-3rxd krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:44.774212 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:45.090255 21286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:45.090579 21286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:45.090821 21286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:45.098065 21286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:45.098206 21286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:45.098312 21286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:45.098400 21286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:45.098487 21286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:45.108121 21286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:45.109385 21286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:45.110702 21286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:45.122840 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:45.122907 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:45.122953 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:45.123602 21286 server_base.cc:1034] running on GCE node
I20260619 01:04:45.124346 21286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:45.125634 21286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:45.126820 21286 hybrid_clock.cc:648] HybridClock initialized: now 1781831085126781 us; error 44 us; skew 500 ppm
I20260619 01:04:45.127168 21286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:45 dist-test-slave-3rxd krb5kdc[21273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:45.130116 21286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:45.131516 21286 webserver.cc:460] Webserver started at http://127.0.0.1:46633/ using document root <none> and password file <none>
I20260619 01:04:45.132118 21286 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:45.132217 21286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:45.132463 21286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:45.135248 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "c78a420c489a4468b8c07ff365e8ebeb"
format_stamp: "Formatted at 2026-06-19 01:04:45 on dist-test-slave-3rxd"
server_key: "4f936fa5a2209cd2980ec77e7b18250f"
server_key_iv: "ff830edaae4c31cdcec969f96cffdff2"
server_key_version: "encryptionkey@0"
I20260619 01:04:45.135944 21286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "c78a420c489a4468b8c07ff365e8ebeb"
format_stamp: "Formatted at 2026-06-19 01:04:45 on dist-test-slave-3rxd"
server_key: "4f936fa5a2209cd2980ec77e7b18250f"
server_key_iv: "ff830edaae4c31cdcec969f96cffdff2"
server_key_version: "encryptionkey@0"
I20260619 01:04:45.140638 21286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 01:04:45.143764 21301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:45.144953 21286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:04:45.145097 21286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "c78a420c489a4468b8c07ff365e8ebeb"
format_stamp: "Formatted at 2026-06-19 01:04:45 on dist-test-slave-3rxd"
server_key: "4f936fa5a2209cd2980ec77e7b18250f"
server_key_iv: "ff830edaae4c31cdcec969f96cffdff2"
server_key_version: "encryptionkey@0"
I20260619 01:04:45.145248 21286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:45.150430 21286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:45.154062 21286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:45.154294 21286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:45.171797 21286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46477
I20260619 01:04:45.171834 21352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46477 every 8 connection(s)
I20260619 01:04:45.173184 21286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:45.175203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21286
I20260619 01:04:45.175392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:45.175846 388 external_mini_cluster.cc:1428] Setting key 65b9458f880ab6f8b224ed5451320f25
I20260619 01:04:45.177301 21353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:45.184244 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:45.177661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36713 (local address 127.0.0.1:46477)
0619 01:04:45.178450 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:45.178464 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:45.178518 (+ 54us) server_negotiation.cc:407] Connection header received
0619 01:04:45.180473 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:45.180500 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:45.180948 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:45.181387 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:45.182350 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:45.182359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:45.182375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:45.182392 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:45.182482 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:45.182507 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:45.182724 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:45.182861 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 01:04:45.183059 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":325,"threads_started":1}
I20260619 01:04:45.187808 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:45.191632 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb: Bootstrap starting.
I20260619 01:04:45.195426 21353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:45.195480 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:45.193017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54785 (local address 127.0.0.1:46477)
0619 01:04:45.193202 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:45.193213 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:45.193537 (+ 324us) server_negotiation.cc:407] Connection header received
0619 01:04:45.193665 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:45.193670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:45.193733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:45.193846 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:45.194763 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:45.194768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:45.194772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:45.194789 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:45.194832 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:45.194849 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:45.194960 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:45.195107 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 01:04:45.195163 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:04:45.196707 21353 log.cc:824] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:45.199456 21353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb: No bootstrap required, opened a new log
I20260619 01:04:45.206645 21353 raft_consensus.cc:348] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260619 01:04:45.207082 21353 raft_consensus.cc:374] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:45.207177 21353 raft_consensus.cc:729] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78a420c489a4468b8c07ff365e8ebeb, State: Initialized, Role: FOLLOWER
I20260619 01:04:45.207746 21353 consensus_queue.cc:260] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260619 01:04:45.207968 21353 raft_consensus.cc:388] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:45.208117 21353 raft_consensus.cc:482] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:45.208284 21353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:45.210520 21353 raft_consensus.cc:504] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260619 01:04:45.211026 21353 leader_election.cc:302] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c78a420c489a4468b8c07ff365e8ebeb; no voters:
I20260619 01:04:45.211532 21353 leader_election.cc:288] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:45.211685 21363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:45.212375 21363 raft_consensus.cc:686] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [term 1 LEADER]: Becoming Leader. State: Replica: c78a420c489a4468b8c07ff365e8ebeb, State: Running, Role: LEADER
I20260619 01:04:45.212888 21363 consensus_queue.cc:237] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260619 01:04:45.213712 21353 sys_catalog.cc:564] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:45.215821 21365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c78a420c489a4468b8c07ff365e8ebeb. Latest consensus state: current_term: 1 leader_uuid: "c78a420c489a4468b8c07ff365e8ebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260619 01:04:45.216158 21365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:45.217671 21364 sys_catalog.cc:455] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c78a420c489a4468b8c07ff365e8ebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c78a420c489a4468b8c07ff365e8ebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260619 01:04:45.217922 21364 sys_catalog.cc:458] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:45.218277 21370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:45.223546 21370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:45.232448 21370 catalog_manager.cc:1347] Generated new cluster ID: cb519d5a605e48a4b1910cc7894ac50f
I20260619 01:04:45.232558 21370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:45.246697 21370 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:45.247094 21370 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:45.251869 21370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c78a420c489a4468b8c07ff365e8ebeb: Generated new TSK 0
I20260619 01:04:45.252549 21370 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:45.258567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21286
W20260619 01:04:45.259542 21348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36713
W20260619 01:04:45.260205 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21286: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:45.260345 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21286
2026-06-19T01:04:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1800 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:45 dist-test-slave-3rxd krb5kdc[21384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:45 dist-test-slave-3rxd krb5kdc[21384](info): set up 2 sockets
Jun 19 01:04:45 dist-test-slave-3rxd krb5kdc[21384](info): commencing operation
krb5kdc: starting...
W20260619 01:04:46.359161 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.006s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:46 dist-test-slave-3rxd krb5kdc[21384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:46.589748 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:46.903229 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:46.903512 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:46.903676 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:46.910915 21397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:46.911027 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:46.911110 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:46.911211 21397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:46.911298 21397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:46.920954 21397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:46.922222 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:46.923573 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:46.935334 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:46.935374 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:46.935421 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:46.940200 21397 server_base.cc:1034] running on GCE node
I20260619 01:04:46.940922 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:46.942286 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:46.943483 21397 hybrid_clock.cc:648] HybridClock initialized: now 1781831086943432 us; error 46 us; skew 500 ppm
I20260619 01:04:46.943827 21397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:46 dist-test-slave-3rxd krb5kdc[21384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:46.953279 21397 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:46.954916 21397 webserver.cc:460] Webserver started at http://127.0.0.1:32831/ using document root <none> and password file <none>
I20260619 01:04:46.955657 21397 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:46.955793 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:46.956110 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:46.959224 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "7f7050b0b8ad4de6a7303e3768fc134f"
format_stamp: "Formatted at 2026-06-19 01:04:46 on dist-test-slave-3rxd"
server_key: "6b91421516a735a22e74403ea4dbcf5f"
server_key_iv: "1f101e83fc9915d16c9e290d61880c74"
server_key_version: "encryptionkey@0"
I20260619 01:04:46.959949 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "7f7050b0b8ad4de6a7303e3768fc134f"
format_stamp: "Formatted at 2026-06-19 01:04:46 on dist-test-slave-3rxd"
server_key: "6b91421516a735a22e74403ea4dbcf5f"
server_key_iv: "1f101e83fc9915d16c9e290d61880c74"
server_key_version: "encryptionkey@0"
I20260619 01:04:46.964854 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:04:46.968290 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:46.969648 21397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 01:04:46.969835 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "7f7050b0b8ad4de6a7303e3768fc134f"
format_stamp: "Formatted at 2026-06-19 01:04:46 on dist-test-slave-3rxd"
server_key: "6b91421516a735a22e74403ea4dbcf5f"
server_key_iv: "1f101e83fc9915d16c9e290d61880c74"
server_key_version: "encryptionkey@0"
I20260619 01:04:46.970041 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:46.990435 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:46.994367 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:46.994647 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:47.012441 21397 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41223
I20260619 01:04:47.012485 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41223 every 8 connection(s)
I20260619 01:04:47.013716 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:47.017220 21464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:47.021660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21397
I20260619 01:04:47.021847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:47.022238 388 external_mini_cluster.cc:1428] Setting key 41bb683f3c8d1f88045e6a148ef1e575
I20260619 01:04:47.029043 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: Bootstrap starting.
I20260619 01:04:47.032577 21464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:47.033553 21464 log.cc:824] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:47.036105 21464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: No bootstrap required, opened a new log
Jun 19 01:04:47 dist-test-slave-3rxd krb5kdc[21384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:04:47.043821 21464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } }
I20260619 01:04:47.044171 21464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:47.044288 21464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7050b0b8ad4de6a7303e3768fc134f, State: Initialized, Role: FOLLOWER
I20260619 01:04:47.044723 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:47.024027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:34125 (local address 127.0.0.1:41223)
0619 01:04:47.024784 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:47.024799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:47.024849 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:04:47.027408 (+ 2559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:47.027436 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:47.027919 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:47.028350 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:47.029580 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:47.031583 (+ 2003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:47.032805 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:47.033451 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:47.033638 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:47.037629 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:47.037636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:47.037649 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:47.037684 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:47.040830 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:47.041559 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:47.041565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:47.041570 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:47.041721 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:47.042168 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:47.042172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:47.042174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:47.042620 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:47.042964 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:47.043379 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 01:04:47.043658 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":305,"threads_started":1}
I20260619 01:04:47.044847 21464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } }
I20260619 01:04:47.045121 21464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:47.045264 21464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:47.045451 21464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:47.047966 21464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } }
I20260619 01:04:47.048368 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:47.048448 21464 leader_election.cc:302] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f7050b0b8ad4de6a7303e3768fc134f; no voters:
I20260619 01:04:47.049011 21464 leader_election.cc:288] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:47.049130 21469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:47.050930 21464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:47.054493 21469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [term 1 LEADER]: Becoming Leader. State: Replica: 7f7050b0b8ad4de6a7303e3768fc134f, State: Running, Role: LEADER
I20260619 01:04:47.055260 21469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } }
I20260619 01:04:47.058409 21470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } } }
I20260619 01:04:47.059062 21471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f7050b0b8ad4de6a7303e3768fc134f. Latest consensus state: current_term: 1 leader_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f7050b0b8ad4de6a7303e3768fc134f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41223 } } }
I20260619 01:04:47.059389 21470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:47.059921 21471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f [sys.catalog]: This master's current role is: LEADER
W20260619 01:04:47.073311 21487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:04:47.073437 21487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:04:47.073577 21489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:47.075762 21489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:47.084103 21467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:47.058875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:35461 (local address 127.0.0.1:41223)
0619 01:04:47.059070 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:47.059078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:47.068363 (+ 9285us) server_negotiation.cc:407] Connection header received
0619 01:04:47.068558 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:47.068564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:47.068638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:47.068750 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:47.070631 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:47.071873 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:47.073753 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:47.074529 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:47.074659 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:47.077696 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:47.077702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:47.077707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:47.077753 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:47.080880 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:47.081646 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:47.081651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:47.081657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:47.081789 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:47.082363 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:47.082368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:47.082372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:47.082801 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:47.083061 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:47.083557 (+ 496us) server_negotiation.cc:299] Negotiation successful
0619 01:04:47.083776 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:04:47.084501 21489 catalog_manager.cc:1347] Generated new cluster ID: 465015708264426b97e34eff005d3453
I20260619 01:04:47.085109 21489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:47.108745 21489 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:47.109930 21489 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:47.121663 21489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f7050b0b8ad4de6a7303e3768fc134f: Generated new TSK 0
I20260619 01:04:47.122404 21489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:47.129154 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21397
I20260619 01:04:47.130599 21459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:47.220517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21397
W20260619 01:04:47.238291 21382 connection.cc:535] client connection to 127.0.0.1:41223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41223): Connection reset by peer (error 104)
2026-06-19T01:04:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:47 dist-test-slave-3rxd krb5kdc[21496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:47 dist-test-slave-3rxd krb5kdc[21496](info): set up 2 sockets
Jun 19 01:04:47 dist-test-slave-3rxd krb5kdc[21496](info): commencing operation
krb5kdc: starting...
W20260619 01:04:48.311117 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:48 dist-test-slave-3rxd krb5kdc[21496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:48.542088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39645
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:48.860088 21509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:48.860414 21509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:48.860575 21509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:48.867705 21509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:48.867851 21509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:48.867964 21509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:48.868079 21509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:48.868160 21509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:48.877755 21509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:48.879027 21509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:48.880261 21509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:48.891875 21514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:48.891844 21517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:48.891804 21515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:48.892828 21509 server_base.cc:1034] running on GCE node
I20260619 01:04:48.893615 21509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:48.895048 21509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:48.896270 21509 hybrid_clock.cc:648] HybridClock initialized: now 1781831088896242 us; error 42 us; skew 500 ppm
I20260619 01:04:48.896600 21509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:48 dist-test-slave-3rxd krb5kdc[21496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:48.899577 21509 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:48.900995 21509 webserver.cc:460] Webserver started at http://127.0.0.1:41287/ using document root <none> and password file <none>
I20260619 01:04:48.901665 21509 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:48.901821 21509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:48.902141 21509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:48.905078 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f"
format_stamp: "Formatted at 2026-06-19 01:04:48 on dist-test-slave-3rxd"
server_key: "c7d2b6ef2016a8bd5e4173a1695d36f3"
server_key_iv: "a574ddbb58bc8dfc47f1a2cba27fc9b2"
server_key_version: "encryptionkey@0"
I20260619 01:04:48.905871 21509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f"
format_stamp: "Formatted at 2026-06-19 01:04:48 on dist-test-slave-3rxd"
server_key: "c7d2b6ef2016a8bd5e4173a1695d36f3"
server_key_iv: "a574ddbb58bc8dfc47f1a2cba27fc9b2"
server_key_version: "encryptionkey@0"
I20260619 01:04:48.910956 21509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 01:04:48.914105 21524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:48.915397 21509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:04:48.915596 21509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f"
format_stamp: "Formatted at 2026-06-19 01:04:48 on dist-test-slave-3rxd"
server_key: "c7d2b6ef2016a8bd5e4173a1695d36f3"
server_key_iv: "a574ddbb58bc8dfc47f1a2cba27fc9b2"
server_key_version: "encryptionkey@0"
I20260619 01:04:48.915792 21509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:48.927634 21509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:48.931488 21509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:48.931768 21509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:48.949712 21509 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39645
I20260619 01:04:48.949738 21575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39645 every 8 connection(s)
I20260619 01:04:48.951160 21509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:48.954857 21576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:48.954885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21509
I20260619 01:04:48.955066 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:48.955636 388 external_mini_cluster.cc:1428] Setting key edf89cc50a3c8297746b598b43771cd9
I20260619 01:04:48.968452 21576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f: Bootstrap starting.
I20260619 01:04:48.968811 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:48.957367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:46221 (local address 127.0.0.1:39645)
0619 01:04:48.958191 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:48.958205 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:48.958247 (+ 42us) server_negotiation.cc:407] Connection header received
0619 01:04:48.959863 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:48.959887 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:48.960267 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:48.960610 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:48.961212 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:48.962511 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:48.964405 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:48.965059 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:48.965183 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:48.966482 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:48.966492 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:48.966505 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:48.967093 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:48.967165 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:48.967188 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:48.967299 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:48.967618 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 01:04:48.967805 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":396,"threads_started":1}
I20260619 01:04:48.971645 21576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:48.972538 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:48.972708 21576 log.cc:824] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:48.975854 21576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f: No bootstrap required, opened a new log
I20260619 01:04:48.984648 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:48.977993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:33603 (local address 127.0.0.1:39645)
0619 01:04:48.978182 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:48.978188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:48.978604 (+ 416us) server_negotiation.cc:407] Connection header received
0619 01:04:48.978778 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:48.978784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:48.978870 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:48.979014 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:48.979475 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:48.980623 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:48.981925 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:48.982636 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:48.982790 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:48.983852 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:48.983858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:48.983864 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:48.983890 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:48.983940 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:48.983959 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:48.984074 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:48.984244 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:04:48.984298 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 01:04:48.986224 21576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } }
I20260619 01:04:48.986588 21576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:48.986657 21576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45899ab0b9a44f279b4ec0b5d98fcf4f, State: Initialized, Role: FOLLOWER
I20260619 01:04:48.987214 21576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } }
I20260619 01:04:48.987417 21576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:48.987524 21576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:48.987711 21576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:48.989941 21576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } }
I20260619 01:04:48.990551 21576 leader_election.cc:302] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45899ab0b9a44f279b4ec0b5d98fcf4f; no voters:
I20260619 01:04:48.991173 21576 leader_election.cc:288] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:48.991299 21586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:48.992070 21586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [term 1 LEADER]: Becoming Leader. State: Replica: 45899ab0b9a44f279b4ec0b5d98fcf4f, State: Running, Role: LEADER
I20260619 01:04:48.992615 21586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } }
I20260619 01:04:48.993394 21576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:48.995779 21588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45899ab0b9a44f279b4ec0b5d98fcf4f. Latest consensus state: current_term: 1 leader_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } } }
I20260619 01:04:48.996150 21588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:48.995872 21587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45899ab0b9a44f279b4ec0b5d98fcf4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39645 } } }
I20260619 01:04:48.996284 21587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:48.999336 21592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:49.004468 21592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:49.013931 21592 catalog_manager.cc:1347] Generated new cluster ID: d09d1a9983ee453dab202871d3cacf05
I20260619 01:04:49.014056 21592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:49.028716 21592 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:49.030017 21592 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:49.045677 21592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45899ab0b9a44f279b4ec0b5d98fcf4f: Generated new TSK 0
I20260619 01:04:49.046783 21592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:49.056746 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21509
W20260619 01:04:49.057945 21571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:46221
W20260619 01:04:49.058682 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21509: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:49.058866 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21509
W20260619 01:04:49.075690 21494 connection.cc:535] client connection to 127.0.0.1:39645 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39645): Connection reset by peer (error 104)
2026-06-19T01:04:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1837 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:49 dist-test-slave-3rxd krb5kdc[21607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:49 dist-test-slave-3rxd krb5kdc[21607](info): set up 2 sockets
Jun 19 01:04:49 dist-test-slave-3rxd krb5kdc[21607](info): commencing operation
krb5kdc: starting...
W20260619 01:04:50.159201 388 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:50 dist-test-slave-3rxd krb5kdc[21607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:50.400223 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33495
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:50.716714 21620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:50.717051 21620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:50.717231 21620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:50.724311 21620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:50.724470 21620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:50.724586 21620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:50.724682 21620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:50.724749 21620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:50.734331 21620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:50.735648 21620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:50.736974 21620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:50.748486 21626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:50.748615 21625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:50.748529 21628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:50.749379 21620 server_base.cc:1034] running on GCE node
I20260619 01:04:50.750155 21620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:50.751610 21620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:50.752807 21620 hybrid_clock.cc:648] HybridClock initialized: now 1781831090752746 us; error 52 us; skew 500 ppm
I20260619 01:04:50.753157 21620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:50 dist-test-slave-3rxd krb5kdc[21607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:50.756255 21620 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:50.757750 21620 webserver.cc:460] Webserver started at http://127.0.0.1:34663/ using document root <none> and password file <none>
I20260619 01:04:50.758430 21620 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:50.758580 21620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:50.758949 21620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:50.761876 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "5082d066074842808ccb5b51b2482660"
format_stamp: "Formatted at 2026-06-19 01:04:50 on dist-test-slave-3rxd"
server_key: "dd1787e721457dfe2781a2376aed2e8d"
server_key_iv: "45f062f582872fcc293c0524dcdfd108"
server_key_version: "encryptionkey@0"
I20260619 01:04:50.762678 21620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "5082d066074842808ccb5b51b2482660"
format_stamp: "Formatted at 2026-06-19 01:04:50 on dist-test-slave-3rxd"
server_key: "dd1787e721457dfe2781a2376aed2e8d"
server_key_iv: "45f062f582872fcc293c0524dcdfd108"
server_key_version: "encryptionkey@0"
I20260619 01:04:50.767869 21620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:04:50.771270 21635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:50.772593 21620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 01:04:50.772781 21620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "5082d066074842808ccb5b51b2482660"
format_stamp: "Formatted at 2026-06-19 01:04:50 on dist-test-slave-3rxd"
server_key: "dd1787e721457dfe2781a2376aed2e8d"
server_key_iv: "45f062f582872fcc293c0524dcdfd108"
server_key_version: "encryptionkey@0"
I20260619 01:04:50.772984 21620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:50.778883 21620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:50.782837 21620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:50.783128 21620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:50.801280 21620 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33495
I20260619 01:04:50.801321 21686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33495 every 8 connection(s)
I20260619 01:04:50.802676 21620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:50.803231 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21620
I20260619 01:04:50.803454 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:50.804050 388 external_mini_cluster.cc:1428] Setting key f73dadcd0b6f57d40dab881d40c704a7
I20260619 01:04:50.807395 21687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:50.811859 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:50.805877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:53129 (local address 127.0.0.1:33495)
0619 01:04:50.806635 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:50.806646 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:50.806686 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:04:50.808475 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:50.808497 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:50.808818 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:50.809196 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:50.810260 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:50.810266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:50.810278 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:50.810297 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:50.810372 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:50.810392 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:50.810638 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:50.810817 (+ 179us) server_negotiation.cc:299] Negotiation successful
0619 01:04:50.811002 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":315,"threads_started":1}
I20260619 01:04:50.815523 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:50.818928 21687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660: Bootstrap starting.
I20260619 01:04:50.822407 21690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:50.819893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:35003 (local address 127.0.0.1:33495)
0619 01:04:50.820117 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:50.820122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:50.820473 (+ 351us) server_negotiation.cc:407] Connection header received
0619 01:04:50.820647 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:50.820651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:50.820697 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:50.820779 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:50.821735 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:50.821740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:50.821743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:50.821757 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:50.821793 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:50.821806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:50.821895 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:50.822046 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 01:04:50.822093 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:04:50.822865 21687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:50.823906 21687 log.cc:824] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:50.826408 21687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660: No bootstrap required, opened a new log
I20260619 01:04:50.833559 21687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260619 01:04:50.833948 21687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:50.834067 21687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5082d066074842808ccb5b51b2482660, State: Initialized, Role: FOLLOWER
I20260619 01:04:50.834620 21687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260619 01:04:50.834908 21687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:50.835048 21687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:50.835230 21687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:50.837509 21687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260619 01:04:50.838002 21687 leader_election.cc:302] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5082d066074842808ccb5b51b2482660; no voters:
I20260619 01:04:50.838599 21687 leader_election.cc:288] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:50.838758 21697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:50.839509 21697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [term 1 LEADER]: Becoming Leader. State: Replica: 5082d066074842808ccb5b51b2482660, State: Running, Role: LEADER
I20260619 01:04:50.839978 21697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } }
I20260619 01:04:50.840726 21687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:50.842656 21698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5082d066074842808ccb5b51b2482660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } } }
I20260619 01:04:50.842918 21699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5082d066074842808ccb5b51b2482660. Latest consensus state: current_term: 1 leader_uuid: "5082d066074842808ccb5b51b2482660" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5082d066074842808ccb5b51b2482660" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33495 } } }
I20260619 01:04:50.843236 21698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:50.843235 21699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:50.844344 21704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:50.850049 21704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:50.859369 21704 catalog_manager.cc:1347] Generated new cluster ID: 4bc653c9c6ea47908df544f8d407af70
I20260619 01:04:50.859551 21704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:50.871198 21704 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:50.871608 21704 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:50.880216 21704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5082d066074842808ccb5b51b2482660: Generated new TSK 0
I20260619 01:04:50.880947 21704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:50.890849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21620
W20260619 01:04:50.891911 21682 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:53129
W20260619 01:04:50.892529 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21620: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:50.892674 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21620
2026-06-19T01:04:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1833 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:50 dist-test-slave-3rxd krb5kdc[21718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:50 dist-test-slave-3rxd krb5kdc[21718](info): set up 2 sockets
Jun 19 01:04:50 dist-test-slave-3rxd krb5kdc[21718](info): commencing operation
krb5kdc: starting...
W20260619 01:04:51.983060 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:52.229347 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:52.546229 21731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:52.546527 21731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:52.546687 21731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:52.553637 21731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:52.553735 21731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:52.553781 21731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:52.553860 21731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:52.553942 21731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:52.563390 21731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:52.564690 21731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:52.565948 21731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:52.577461 21736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:52.577522 21739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:52.577639 21737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:52.578630 21731 server_base.cc:1034] running on GCE node
I20260619 01:04:52.579511 21731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:52.580859 21731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:52.582062 21731 hybrid_clock.cc:648] HybridClock initialized: now 1781831092582017 us; error 29 us; skew 500 ppm
I20260619 01:04:52.582367 21731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:52.589826 21731 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:52.591470 21731 webserver.cc:460] Webserver started at http://127.0.0.1:33945/ using document root <none> and password file <none>
I20260619 01:04:52.592267 21731 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:52.592401 21731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:52.592720 21731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:52.595814 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "035b7c0636244f099a4ca43a0b70ff23"
format_stamp: "Formatted at 2026-06-19 01:04:52 on dist-test-slave-3rxd"
server_key: "984df1b970b0c534ab50cf9741195cf9"
server_key_iv: "e3751a2d86e8d35c4d7bf2a1232505f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:52.596633 21731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "035b7c0636244f099a4ca43a0b70ff23"
format_stamp: "Formatted at 2026-06-19 01:04:52 on dist-test-slave-3rxd"
server_key: "984df1b970b0c534ab50cf9741195cf9"
server_key_iv: "e3751a2d86e8d35c4d7bf2a1232505f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:52.601634 21731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 01:04:52.605028 21746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:52.606253 21731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:04:52.606431 21731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "035b7c0636244f099a4ca43a0b70ff23"
format_stamp: "Formatted at 2026-06-19 01:04:52 on dist-test-slave-3rxd"
server_key: "984df1b970b0c534ab50cf9741195cf9"
server_key_iv: "e3751a2d86e8d35c4d7bf2a1232505f4"
server_key_version: "encryptionkey@0"
I20260619 01:04:52.606621 21731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:52.633432 21731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:52.637571 21731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:52.637872 21731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:52.656286 21731 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44221
I20260619 01:04:52.656319 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44221 every 8 connection(s)
I20260619 01:04:52.657691 21731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:52.661449 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:52.663440 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21731
I20260619 01:04:52.663638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:52.664067 388 external_mini_cluster.cc:1428] Setting key b267db935a9aef1e817ae5bd6b3376d3
I20260619 01:04:52.672426 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23: Bootstrap starting.
I20260619 01:04:52.675500 21798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:52.676466 21798 log.cc:824] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 01:04:52.679618 21798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23: No bootstrap required, opened a new log
I20260619 01:04:52.686108 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:52.665920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:47491 (local address 127.0.0.1:44221)
0619 01:04:52.666652 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:52.666664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:52.666748 (+ 84us) server_negotiation.cc:407] Connection header received
0619 01:04:52.668411 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:52.668431 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:52.668819 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:52.669182 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:52.670614 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:52.671954 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:52.673287 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:52.673955 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:52.674151 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:52.678596 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:52.678604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:52.678617 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:52.678650 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:52.682100 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:52.682977 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:52.682982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:52.682988 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:52.683148 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:52.683615 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:52.683618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:52.683620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:52.684084 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:52.684302 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:52.684817 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 01:04:52.685126 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":283,"threads_started":1}
I20260619 01:04:52.688609 21798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260619 01:04:52.689013 21798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:52.689141 21798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035b7c0636244f099a4ca43a0b70ff23, State: Initialized, Role: FOLLOWER
I20260619 01:04:52.689808 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:52.689790 21798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260619 01:04:52.690076 21798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:52.690200 21798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:52.690356 21798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:52.693678 21798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260619 01:04:52.694293 21798 leader_election.cc:302] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 035b7c0636244f099a4ca43a0b70ff23; no voters:
I20260619 01:04:52.695013 21798 leader_election.cc:288] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:52.695168 21807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:52.696287 21807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [term 1 LEADER]: Becoming Leader. State: Replica: 035b7c0636244f099a4ca43a0b70ff23, State: Running, Role: LEADER
I20260619 01:04:52.697120 21807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } }
I20260619 01:04:52.700186 21798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:52.708069 21808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 035b7c0636244f099a4ca43a0b70ff23. Latest consensus state: current_term: 1 leader_uuid: "035b7c0636244f099a4ca43a0b70ff23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } } }
I20260619 01:04:52.708621 21808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:52.709273 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "035b7c0636244f099a4ca43a0b70ff23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "035b7c0636244f099a4ca43a0b70ff23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44221 } } }
I20260619 01:04:52.709614 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:52.716292 21820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:52.723125 21820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:52.725719 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:52.695311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:40227 (local address 127.0.0.1:44221)
0619 01:04:52.695521 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:52.695530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:52.701630 (+ 6100us) server_negotiation.cc:407] Connection header received
0619 01:04:52.701849 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:52.701863 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:52.701971 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:52.702121 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:52.704689 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:52.706360 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:52.712409 (+ 6049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:52.713364 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:52.713503 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:52.717513 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:52.717518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:52.717521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:52.717551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:52.721599 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:52.722582 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:52.722589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:52.722593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:52.722761 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:52.723359 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:52.723364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:52.723367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:52.723797 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:52.724978 (+ 1181us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:52.725164 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 01:04:52.725341 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:04:52.740947 21820 catalog_manager.cc:1347] Generated new cluster ID: 5ab7d2c99b4f4ef0b5d3fd95352583a6
I20260619 01:04:52.741056 21820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:52.768358 21820 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:52.769703 21820 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:52.779909 21820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 035b7c0636244f099a4ca43a0b70ff23: Generated new TSK 0
I20260619 01:04:52.780699 21820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:52.793084 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21731
I20260619 01:04:52.794211 21793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:52.885846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21731
W20260619 01:04:52.903192 21716 connection.cc:535] client connection to 127.0.0.1:44221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44221): Connection reset by peer (error 104)
2026-06-19T01:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21831](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21831](info): set up 2 sockets
Jun 19 01:04:52 dist-test-slave-3rxd krb5kdc[21831](info): commencing operation
krb5kdc: starting...
W20260619 01:04:53.994943 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:54 dist-test-slave-3rxd krb5kdc[21831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:54.229628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46347
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:54.547492 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:54.547806 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:54.547979 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:54.555608 21844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:54.555766 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:54.555878 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:54.555987 21844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:54.556072 21844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:54.565845 21844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46347
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:54.567198 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:54.568576 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:54.580210 21849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:54.580174 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:54.580245 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:54.581338 21844 server_base.cc:1034] running on GCE node
I20260619 01:04:54.582113 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:54.583503 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:54.584728 21844 hybrid_clock.cc:648] HybridClock initialized: now 1781831094584702 us; error 54 us; skew 500 ppm
I20260619 01:04:54.585136 21844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:54 dist-test-slave-3rxd krb5kdc[21831](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:54.589854 21844 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:54.591377 21844 webserver.cc:460] Webserver started at http://127.0.0.1:39991/ using document root <none> and password file <none>
I20260619 01:04:54.592061 21844 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:54.592208 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:54.592520 21844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:54.595649 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "857e7f21a0e34190ad8e39621717ed06"
format_stamp: "Formatted at 2026-06-19 01:04:54 on dist-test-slave-3rxd"
server_key: "0365199b6e5c8c4579cda936ffd806bb"
server_key_iv: "4c69b2f1641ea68a97f6f10060101f73"
server_key_version: "encryptionkey@0"
I20260619 01:04:54.596433 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "857e7f21a0e34190ad8e39621717ed06"
format_stamp: "Formatted at 2026-06-19 01:04:54 on dist-test-slave-3rxd"
server_key: "0365199b6e5c8c4579cda936ffd806bb"
server_key_iv: "4c69b2f1641ea68a97f6f10060101f73"
server_key_version: "encryptionkey@0"
I20260619 01:04:54.601632 21844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 01:04:54.604890 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:54.606216 21844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:04:54.606392 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "857e7f21a0e34190ad8e39621717ed06"
format_stamp: "Formatted at 2026-06-19 01:04:54 on dist-test-slave-3rxd"
server_key: "0365199b6e5c8c4579cda936ffd806bb"
server_key_iv: "4c69b2f1641ea68a97f6f10060101f73"
server_key_version: "encryptionkey@0"
I20260619 01:04:54.606580 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:54.627856 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:54.632198 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:54.632483 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:54.650447 21844 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46347
I20260619 01:04:54.650484 21910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46347 every 8 connection(s)
I20260619 01:04:54.651821 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:54.655426 21911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:54.660153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21844
I20260619 01:04:54.660320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:54.660861 388 external_mini_cluster.cc:1428] Setting key 294f33b14476a66f53e7831cd5f22c91
I20260619 01:04:54.666502 21911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06: Bootstrap starting.
I20260619 01:04:54.669526 21911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:54.670516 21911 log.cc:824] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:54.673866 21911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06: No bootstrap required, opened a new log
I20260619 01:04:54.674103 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:54.662690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:45943 (local address 127.0.0.1:46347)
0619 01:04:54.663430 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:54.663441 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:54.663480 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:04:54.665161 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:54.665182 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:54.665628 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:54.666097 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:54.666602 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:54.667973 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:54.669248 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:54.669986 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:54.670190 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:54.671521 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:54.671528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:54.671540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:54.672157 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:54.672246 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:54.672276 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:54.672438 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:54.672673 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 01:04:54.672834 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":315,"threads_started":1}
I20260619 01:04:54.677973 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:54.681876 21911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } }
I20260619 01:04:54.682461 21911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:54.682595 21911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 857e7f21a0e34190ad8e39621717ed06, State: Initialized, Role: FOLLOWER
I20260619 01:04:54.683238 21911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } }
I20260619 01:04:54.683488 21911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:54.683629 21911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:54.683822 21911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:54.685873 21911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } }
I20260619 01:04:54.686359 21911 leader_election.cc:302] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 857e7f21a0e34190ad8e39621717ed06; no voters:
I20260619 01:04:54.687050 21911 leader_election.cc:288] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:54.687131 21921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:54.688652 21914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:54.682254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:35093 (local address 127.0.0.1:46347)
0619 01:04:54.682434 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:54.682441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:54.682822 (+ 381us) server_negotiation.cc:407] Connection header received
0619 01:04:54.683014 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:54.683018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:54.683074 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:54.683173 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:54.683708 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:54.684490 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:54.685766 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:54.686282 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:54.686435 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:54.687772 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:54.687782 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:54.687786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:54.687806 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:54.687847 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:54.687867 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:54.687960 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:54.688363 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 01:04:54.688410 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:04:54.689499 21911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:54.690971 21921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [term 1 LEADER]: Becoming Leader. State: Replica: 857e7f21a0e34190ad8e39621717ed06, State: Running, Role: LEADER
I20260619 01:04:54.691758 21921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } }
I20260619 01:04:54.694778 21923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "857e7f21a0e34190ad8e39621717ed06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } } }
I20260619 01:04:54.694880 21922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 857e7f21a0e34190ad8e39621717ed06. Latest consensus state: current_term: 1 leader_uuid: "857e7f21a0e34190ad8e39621717ed06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "857e7f21a0e34190ad8e39621717ed06" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46347 } } }
I20260619 01:04:54.695250 21922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:54.695227 21923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:54.699584 21931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:54.704365 21931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:54.713650 21931 catalog_manager.cc:1347] Generated new cluster ID: 6db80aedeecd48609040c8d4615264f2
I20260619 01:04:54.713768 21931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:54.727308 21931 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:54.728505 21931 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:54.736145 21931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 857e7f21a0e34190ad8e39621717ed06: Generated new TSK 0
I20260619 01:04:54.737146 21931 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:54.754935 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21844
W20260619 01:04:54.755978 21906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:45943
W20260619 01:04:54.756788 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21844: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:54.756932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21844
W20260619 01:04:54.773439 21829 connection.cc:535] client connection to 127.0.0.1:46347 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46347): Connection reset by peer (error 104)
2026-06-19T01:04:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:54 dist-test-slave-3rxd krb5kdc[21942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:54 dist-test-slave-3rxd krb5kdc[21942](info): set up 2 sockets
Jun 19 01:04:54 dist-test-slave-3rxd krb5kdc[21942](info): commencing operation
krb5kdc: starting...
W20260619 01:04:55.839432 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:55 dist-test-slave-3rxd krb5kdc[21942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 01:04:56.071451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:56.386758 21955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:56.387001 21955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:56.387147 21955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:56.394115 21955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:56.394248 21955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:56.394346 21955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:56.394446 21955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:56.394529 21955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:56.403930 21955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:56.405043 21955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:56.406234 21955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:56.417744 21960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:56.417640 21961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:56.417713 21963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:56.418457 21955 server_base.cc:1034] running on GCE node
I20260619 01:04:56.419283 21955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:56.420622 21955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:56.421845 21955 hybrid_clock.cc:648] HybridClock initialized: now 1781831096421810 us; error 48 us; skew 500 ppm
I20260619 01:04:56.422164 21955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:56 dist-test-slave-3rxd krb5kdc[21942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:56.425027 21955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 01:04:56.426376 21955 webserver.cc:460] Webserver started at http://127.0.0.1:37979/ using document root <none> and password file <none>
I20260619 01:04:56.427104 21955 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:56.427241 21955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:56.427531 21955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:56.430294 21955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "3e83a5c6e052428d865f54b25ae77551"
format_stamp: "Formatted at 2026-06-19 01:04:56 on dist-test-slave-3rxd"
server_key: "fff87d62f03f2b53ea73156e219575ec"
server_key_iv: "014915ed91ad9d3a3e6e1bc7a3d1ab9e"
server_key_version: "encryptionkey@0"
I20260619 01:04:56.431118 21955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "3e83a5c6e052428d865f54b25ae77551"
format_stamp: "Formatted at 2026-06-19 01:04:56 on dist-test-slave-3rxd"
server_key: "fff87d62f03f2b53ea73156e219575ec"
server_key_iv: "014915ed91ad9d3a3e6e1bc7a3d1ab9e"
server_key_version: "encryptionkey@0"
I20260619 01:04:56.435833 21955 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 01:04:56.438959 21970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:56.440133 21955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 01:04:56.440308 21955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "3e83a5c6e052428d865f54b25ae77551"
format_stamp: "Formatted at 2026-06-19 01:04:56 on dist-test-slave-3rxd"
server_key: "fff87d62f03f2b53ea73156e219575ec"
server_key_iv: "014915ed91ad9d3a3e6e1bc7a3d1ab9e"
server_key_version: "encryptionkey@0"
I20260619 01:04:56.440506 21955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:56.445806 21955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:56.449347 21955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:56.449631 21955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:56.467202 21955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35015
I20260619 01:04:56.467276 22021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35015 every 8 connection(s)
I20260619 01:04:56.468456 21955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:56.471868 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:56.472504 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 21955
I20260619 01:04:56.472685 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:56.473206 388 external_mini_cluster.cc:1428] Setting key d5d25748da150179c0593f440bbf5fc6
I20260619 01:04:56.480690 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:56.474958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:55649 (local address 127.0.0.1:35015)
0619 01:04:56.475710 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:56.475723 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:56.475762 (+ 39us) server_negotiation.cc:407] Connection header received
0619 01:04:56.477725 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:56.477747 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:56.478061 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:56.478361 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:56.479355 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:56.479361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:56.479372 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:56.479384 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:56.479433 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:56.479451 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:56.479604 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:56.479697 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 01:04:56.479883 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":348,"threads_started":1}
I20260619 01:04:56.484143 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:56.484972 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551: Bootstrap starting.
I20260619 01:04:56.487716 22022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:56.488976 22022 log.cc:824] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551: Log is configured to *not* fsync() on all Append() calls
I20260619 01:04:56.490971 22025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:56.488359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.135:43823 (local address 127.0.0.1:35015)
0619 01:04:56.488545 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:56.488552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:56.488887 (+ 335us) server_negotiation.cc:407] Connection header received
0619 01:04:56.489133 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:56.489141 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:56.489197 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:56.489273 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:56.490237 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:56.490243 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:56.490248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:04:56.490265 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:56.490310 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:04:56.490324 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:56.490407 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:56.490560 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 01:04:56.490619 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:04:56.492129 22022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551: No bootstrap required, opened a new log
I20260619 01:04:56.499859 22022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } }
I20260619 01:04:56.500237 22022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:56.500339 22022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e83a5c6e052428d865f54b25ae77551, State: Initialized, Role: FOLLOWER
I20260619 01:04:56.500890 22022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } }
I20260619 01:04:56.501156 22022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:56.501302 22022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:56.501480 22022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:56.503590 22022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } }
I20260619 01:04:56.504127 22022 leader_election.cc:302] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e83a5c6e052428d865f54b25ae77551; no voters:
I20260619 01:04:56.504796 22022 leader_election.cc:288] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:56.504905 22032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:56.505533 22032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [term 1 LEADER]: Becoming Leader. State: Replica: 3e83a5c6e052428d865f54b25ae77551, State: Running, Role: LEADER
I20260619 01:04:56.506124 22032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } }
I20260619 01:04:56.506897 22022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:56.508795 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e83a5c6e052428d865f54b25ae77551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } } }
I20260619 01:04:56.509181 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:56.511061 22034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e83a5c6e052428d865f54b25ae77551. Latest consensus state: current_term: 1 leader_uuid: "3e83a5c6e052428d865f54b25ae77551" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e83a5c6e052428d865f54b25ae77551" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35015 } } }
I20260619 01:04:56.511430 22034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:56.512010 22038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:56.518105 22038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:56.526603 22038 catalog_manager.cc:1347] Generated new cluster ID: 589d168e0c0148c285e231f0927c9572
I20260619 01:04:56.526736 22038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:56.553957 22038 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:56.554399 22038 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:56.573822 22038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e83a5c6e052428d865f54b25ae77551: Generated new TSK 0
I20260619 01:04:56.574802 22038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:56.585407 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21955
W20260619 01:04:56.586395 22017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.135:55649
W20260619 01:04:56.587051 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 21955: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:04:56.587193 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 21955
2026-06-19T01:04:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:56 dist-test-slave-3rxd krb5kdc[22053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:56 dist-test-slave-3rxd krb5kdc[22053](info): set up 2 sockets
Jun 19 01:04:56 dist-test-slave-3rxd krb5kdc[22053](info): commencing operation
krb5kdc: starting...
W20260619 01:04:57.671249 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:57 dist-test-slave-3rxd krb5kdc[22053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:04:57.921631 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42049
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:04:58.238453 22066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:04:58.238757 22066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:04:58.238909 22066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:04:58.245839 22066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:04:58.245949 22066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:04:58.245996 22066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:04:58.246057 22066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:04:58.246094 22066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:04:58.255611 22066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42049
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:04:58.256768 22066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:04:58.258450 22066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:04:58.269759 22071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:58.269821 22072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:04:58.269979 22074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:04:58.270781 22066 server_base.cc:1034] running on GCE node
I20260619 01:04:58.271458 22066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:04:58.272740 22066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:04:58.273974 22066 hybrid_clock.cc:648] HybridClock initialized: now 1781831098273928 us; error 43 us; skew 500 ppm
I20260619 01:04:58.274300 22066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:04:58 dist-test-slave-3rxd krb5kdc[22053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:04:58.281023 22066 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:04:58.282375 22066 webserver.cc:460] Webserver started at http://127.0.97.62:34907/ using document root <none> and password file <none>
I20260619 01:04:58.283095 22066 fs_manager.cc:359] Metadata directory not provided
I20260619 01:04:58.283221 22066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:04:58.283535 22066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:04:58.286244 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "643fb78600a248bdb5ba6821f6176ea5"
format_stamp: "Formatted at 2026-06-19 01:04:58 on dist-test-slave-3rxd"
server_key: "9b6b4328ea92514e3ec8e0df7d708c81"
server_key_iv: "bf1640ad5aaa68b6eefd488c6b574c31"
server_key_version: "encryptionkey@0"
I20260619 01:04:58.287016 22066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "643fb78600a248bdb5ba6821f6176ea5"
format_stamp: "Formatted at 2026-06-19 01:04:58 on dist-test-slave-3rxd"
server_key: "9b6b4328ea92514e3ec8e0df7d708c81"
server_key_iv: "bf1640ad5aaa68b6eefd488c6b574c31"
server_key_version: "encryptionkey@0"
I20260619 01:04:58.291940 22066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:04:58.295318 22081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:04:58.296548 22066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 01:04:58.296733 22066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "643fb78600a248bdb5ba6821f6176ea5"
format_stamp: "Formatted at 2026-06-19 01:04:58 on dist-test-slave-3rxd"
server_key: "9b6b4328ea92514e3ec8e0df7d708c81"
server_key_iv: "bf1640ad5aaa68b6eefd488c6b574c31"
server_key_version: "encryptionkey@0"
I20260619 01:04:58.296974 22066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:04:58.324237 22066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:04:58.328140 22066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:04:58.328434 22066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:04:58.346601 22066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42049
I20260619 01:04:58.346635 22132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42049 every 8 connection(s)
I20260619 01:04:58.347961 22066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:04:58.351599 22133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:04:58.354590 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22066
I20260619 01:04:58.354792 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:04:58.355325 388 external_mini_cluster.cc:1428] Setting key b1416902c0b87b6414e2caf5575aa6ab
I20260619 01:04:58.364354 22133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5: Bootstrap starting.
I20260619 01:04:58.368095 22133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5: Neither blocks nor log segments found. Creating new log.
I20260619 01:04:58.369378 22133 log.cc:824] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5: Log is configured to *not* fsync() on all Append() calls
Jun 19 01:04:58 dist-test-slave-3rxd krb5kdc[22053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:04:58.372360 22133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5: No bootstrap required, opened a new log
I20260619 01:04:58.377735 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:58.357173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45713 (local address 127.0.97.62:42049)
0619 01:04:58.358011 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:58.358022 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:58.358071 (+ 49us) server_negotiation.cc:407] Connection header received
0619 01:04:58.359751 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:58.359773 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:58.360144 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:58.360494 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:58.362175 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:58.363506 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:58.365197 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:58.365842 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:58.365965 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:58.370894 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:58.370902 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:58.370917 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:58.370954 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:58.373811 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:58.374548 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:58.374553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:58.374563 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:58.374780 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:58.375215 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:58.375219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:58.375227 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:58.375710 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:58.376027 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:58.376534 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 01:04:58.376813 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":383,"threads_started":1}
I20260619 01:04:58.379817 22133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } }
I20260619 01:04:58.380249 22133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:04:58.380353 22133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643fb78600a248bdb5ba6821f6176ea5, State: Initialized, Role: FOLLOWER
I20260619 01:04:58.380959 22133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } }
I20260619 01:04:58.381234 22133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:04:58.381369 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:04:58.381392 22133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:04:58.381588 22133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:04:58.383774 22133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } }
I20260619 01:04:58.384286 22133 leader_election.cc:302] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 643fb78600a248bdb5ba6821f6176ea5; no voters:
I20260619 01:04:58.384898 22133 leader_election.cc:288] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:04:58.384986 22142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:04:58.387305 22133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:04:58.391916 22142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 643fb78600a248bdb5ba6821f6176ea5, State: Running, Role: LEADER
I20260619 01:04:58.392554 22142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } }
I20260619 01:04:58.395848 22143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "643fb78600a248bdb5ba6821f6176ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } } }
I20260619 01:04:58.396324 22143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:58.396976 22144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 643fb78600a248bdb5ba6821f6176ea5. Latest consensus state: current_term: 1 leader_uuid: "643fb78600a248bdb5ba6821f6176ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "643fb78600a248bdb5ba6821f6176ea5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42049 } } }
I20260619 01:04:58.397291 22144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5 [sys.catalog]: This master's current role is: LEADER
I20260619 01:04:58.407070 22153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:04:58.410183 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:04:58.388216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50535 (local address 127.0.97.62:42049)
0619 01:04:58.388417 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:04:58.388426 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:04:58.392641 (+ 4215us) server_negotiation.cc:407] Connection header received
0619 01:04:58.392831 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:04:58.392839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:04:58.392926 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:04:58.393036 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:04:58.394633 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:58.395831 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:58.398102 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:04:58.399133 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:04:58.399259 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:04:58.403278 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:04:58.403284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:04:58.403288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:04:58.403325 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:04:58.406297 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:58.407304 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:58.407309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:58.407313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:58.407453 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:04:58.408024 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:04:58.408029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:04:58.408032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:04:58.408587 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:04:58.408877 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 01:04:58.409647 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 01:04:58.409820 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:04:58.413529 22153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:04:58.421955 22153 catalog_manager.cc:1347] Generated new cluster ID: 631e84d6952d4e988af02b86d3f9e1da
I20260619 01:04:58.422098 22153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:04:58.431780 22153 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:04:58.433471 22153 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:04:58.442032 22153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 643fb78600a248bdb5ba6821f6176ea5: Generated new TSK 0
I20260619 01:04:58.442974 22153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:04:58.448290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22066
I20260619 01:04:58.449436 22128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:04:58.538034 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22066
W20260619 01:04:58.554816 22051 connection.cc:535] client connection to 127.0.97.62:42049 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42049): Connection reset by peer (error 104)
2026-06-19T01:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:04:58 dist-test-slave-3rxd krb5kdc[22166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:04:58 dist-test-slave-3rxd krb5kdc[22166](info): set up 2 sockets
Jun 19 01:04:58 dist-test-slave-3rxd krb5kdc[22166](info): commencing operation
krb5kdc: starting...
W20260619 01:04:59.631058 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:04:59 dist-test-slave-3rxd krb5kdc[22166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:04:59.867619 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41641
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:00.196012 22179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:00.196292 22179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:00.196389 22179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:00.203476 22179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:00.203585 22179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:00.203644 22179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:00.203716 22179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:00.203773 22179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:00.213581 22179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:00.214812 22179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:00.216075 22179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:00.228224 22187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:00.228238 22184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:00.228268 22185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:00.228850 22179 server_base.cc:1034] running on GCE node
I20260619 01:05:00.229518 22179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:00.230806 22179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:00.232038 22179 hybrid_clock.cc:648] HybridClock initialized: now 1781831100231973 us; error 42 us; skew 500 ppm
I20260619 01:05:00.232404 22179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:00 dist-test-slave-3rxd krb5kdc[22166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:00.235323 22179 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:00.236760 22179 webserver.cc:460] Webserver started at http://127.0.97.62:37581/ using document root <none> and password file <none>
I20260619 01:05:00.237452 22179 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:00.237608 22179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:00.237919 22179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:00.240778 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "3757e660c550463eac6604668dcc8d3d"
format_stamp: "Formatted at 2026-06-19 01:05:00 on dist-test-slave-3rxd"
server_key: "e7a48a3edeb694b7488cd51e1a423246"
server_key_iv: "e461d763c95dda47d91daa37140c3500"
server_key_version: "encryptionkey@0"
I20260619 01:05:00.241490 22179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "3757e660c550463eac6604668dcc8d3d"
format_stamp: "Formatted at 2026-06-19 01:05:00 on dist-test-slave-3rxd"
server_key: "e7a48a3edeb694b7488cd51e1a423246"
server_key_iv: "e461d763c95dda47d91daa37140c3500"
server_key_version: "encryptionkey@0"
I20260619 01:05:00.246196 22179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:05:00.249610 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:00.250919 22179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:05:00.251096 22179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "3757e660c550463eac6604668dcc8d3d"
format_stamp: "Formatted at 2026-06-19 01:05:00 on dist-test-slave-3rxd"
server_key: "e7a48a3edeb694b7488cd51e1a423246"
server_key_iv: "e461d763c95dda47d91daa37140c3500"
server_key_version: "encryptionkey@0"
I20260619 01:05:00.251281 22179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:00.264674 22179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:00.268262 22179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:00.268556 22179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:00.286996 22179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41641
I20260619 01:05:00.287035 22245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41641 every 8 connection(s)
I20260619 01:05:00.288429 22179 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:00.290664 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22179
I20260619 01:05:00.290879 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:00.291313 388 external_mini_cluster.cc:1428] Setting key cd8ea014f49cbe9d62a6ff343068186c
I20260619 01:05:00.292656 22246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:00.306021 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:00.293116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38137 (local address 127.0.97.62:41641)
0619 01:05:00.293929 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:00.293943 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:00.293994 (+ 51us) server_negotiation.cc:407] Connection header received
0619 01:05:00.295985 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:00.296012 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:00.296592 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:00.297073 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:00.297584 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:00.299642 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:00.300905 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:00.301917 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:00.302088 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:00.303262 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:00.303275 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:00.303298 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:00.304056 (+ 758us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:00.304155 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:00.304193 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:00.304351 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:00.304564 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 01:05:00.304743 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":350,"threads_started":1}
I20260619 01:05:00.307587 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d: Bootstrap starting.
I20260619 01:05:00.309791 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:00.310920 22246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:00.312127 22246 log.cc:824] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:00.315009 22246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d: No bootstrap required, opened a new log
I20260619 01:05:00.321952 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:00.315214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53325 (local address 127.0.97.62:41641)
0619 01:05:00.315384 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:00.315390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:00.315843 (+ 453us) server_negotiation.cc:407] Connection header received
0619 01:05:00.315980 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:00.315985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:00.316061 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:00.316165 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:00.316654 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:00.317849 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:00.319229 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:00.319925 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:00.320074 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:00.321166 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:00.321173 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:00.321178 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:00.321207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:00.321252 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:00.321272 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:00.321385 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:00.321550 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 01:05:00.321605 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:05:00.324589 22246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260619 01:05:00.324927 22246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:05:00.324995 22246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3757e660c550463eac6604668dcc8d3d, State: Initialized, Role: FOLLOWER
I20260619 01:05:00.325562 22246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260619 01:05:00.325769 22246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:05:00.325868 22246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:05:00.325984 22246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:00.328194 22246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260619 01:05:00.328626 22246 leader_election.cc:302] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3757e660c550463eac6604668dcc8d3d; no voters:
I20260619 01:05:00.329218 22246 leader_election.cc:288] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:05:00.329339 22256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:00.330423 22256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [term 1 LEADER]: Becoming Leader. State: Replica: 3757e660c550463eac6604668dcc8d3d, State: Running, Role: LEADER
I20260619 01:05:00.331004 22256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } }
I20260619 01:05:00.331662 22246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:00.333830 22257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3757e660c550463eac6604668dcc8d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } } }
I20260619 01:05:00.334185 22257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:00.336479 22261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:00.336336 22258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3757e660c550463eac6604668dcc8d3d. Latest consensus state: current_term: 1 leader_uuid: "3757e660c550463eac6604668dcc8d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3757e660c550463eac6604668dcc8d3d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41641 } } }
I20260619 01:05:00.336720 22258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:00.341189 22261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:00.348805 22261 catalog_manager.cc:1347] Generated new cluster ID: 491f61abfafe45cabd7e5cd62798fd32
I20260619 01:05:00.348943 22261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:00.360268 22261 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:00.361385 22261 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:00.370471 22261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3757e660c550463eac6604668dcc8d3d: Generated new TSK 0
I20260619 01:05:00.371227 22261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:00.378290 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22179
W20260619 01:05:00.379395 22241 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38137
W20260619 01:05:00.380115 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22179: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:05:00.380257 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22179
W20260619 01:05:00.396696 22164 connection.cc:535] client connection to 127.0.97.62:41641 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:41641): Connection reset by peer (error 104)
2026-06-19T01:05:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:00 dist-test-slave-3rxd krb5kdc[22277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:00 dist-test-slave-3rxd krb5kdc[22277](info): set up 2 sockets
Jun 19 01:05:00 dist-test-slave-3rxd krb5kdc[22277](info): commencing operation
krb5kdc: starting...
W20260619 01:05:01.463234 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:01 dist-test-slave-3rxd krb5kdc[22277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:01.706610 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46059
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:02.033641 22290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:02.034004 22290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:02.034168 22290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:02.041399 22290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:02.041556 22290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:02.041659 22290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:02.041767 22290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:02.041844 22290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:02.051440 22290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:02.052788 22290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:02.054083 22290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:02.066246 22295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:02.066269 22298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:02.066313 22296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:02.067145 22290 server_base.cc:1034] running on GCE node
I20260619 01:05:02.067904 22290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:02.069314 22290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:02.070510 22290 hybrid_clock.cc:648] HybridClock initialized: now 1781831102070451 us; error 52 us; skew 500 ppm
I20260619 01:05:02.070901 22290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:02 dist-test-slave-3rxd krb5kdc[22277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:02.073868 22290 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:02.075501 22290 webserver.cc:460] Webserver started at http://127.0.97.62:34149/ using document root <none> and password file <none>
I20260619 01:05:02.076185 22290 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:02.076326 22290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:02.076628 22290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:02.079631 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "669afdb3d91d45dd9716f3fcb1a85cf9"
format_stamp: "Formatted at 2026-06-19 01:05:02 on dist-test-slave-3rxd"
server_key: "04f9bb4696d64f4cbc2d88d71f57e956"
server_key_iv: "2d857d0e99f9a3957536799db96dc58c"
server_key_version: "encryptionkey@0"
I20260619 01:05:02.080358 22290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "669afdb3d91d45dd9716f3fcb1a85cf9"
format_stamp: "Formatted at 2026-06-19 01:05:02 on dist-test-slave-3rxd"
server_key: "04f9bb4696d64f4cbc2d88d71f57e956"
server_key_iv: "2d857d0e99f9a3957536799db96dc58c"
server_key_version: "encryptionkey@0"
I20260619 01:05:02.085362 22290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 01:05:02.088773 22305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:02.090175 22290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 01:05:02.090358 22290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "669afdb3d91d45dd9716f3fcb1a85cf9"
format_stamp: "Formatted at 2026-06-19 01:05:02 on dist-test-slave-3rxd"
server_key: "04f9bb4696d64f4cbc2d88d71f57e956"
server_key_iv: "2d857d0e99f9a3957536799db96dc58c"
server_key_version: "encryptionkey@0"
I20260619 01:05:02.090565 22290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:02.096258 22290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:02.100317 22290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:02.100615 22290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:02.119625 22290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46059
I20260619 01:05:02.119652 22356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46059 every 8 connection(s)
I20260619 01:05:02.121109 22290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:02.126003 22357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:02.130453 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22290
I20260619 01:05:02.130622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:02.131104 388 external_mini_cluster.cc:1428] Setting key 2ed3916cbcfc65669607a2fd357dc37c
I20260619 01:05:02.137302 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9: Bootstrap starting.
I20260619 01:05:02.139099 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:02.132922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46587 (local address 127.0.97.62:46059)
0619 01:05:02.133732 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:02.133744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:02.133784 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:05:02.135722 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:02.135744 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:02.136088 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:02.136397 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:02.137484 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:02.137491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:02.137504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:02.137518 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:02.137575 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:02.137598 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:02.137758 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:02.137906 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 01:05:02.138125 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":361,"threads_started":1}
I20260619 01:05:02.140867 22357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:02.142155 22357 log.cc:824] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:02.142691 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:02.144831 22357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9: No bootstrap required, opened a new log
I20260619 01:05:02.149401 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:02.146901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41233 (local address 127.0.97.62:46059)
0619 01:05:02.147151 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:02.147160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:02.147531 (+ 371us) server_negotiation.cc:407] Connection header received
0619 01:05:02.147703 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:02.147707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:02.147751 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:02.147845 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:02.148763 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:02.148767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:02.148770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:02.148784 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:02.148817 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:02.148835 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:02.148924 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:02.149066 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 01:05:02.149112 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:05:02.154199 22357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260619 01:05:02.154759 22357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:05:02.154887 22357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669afdb3d91d45dd9716f3fcb1a85cf9, State: Initialized, Role: FOLLOWER
I20260619 01:05:02.155545 22357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260619 01:05:02.155759 22357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:05:02.155851 22357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:05:02.155972 22357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:02.158167 22357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260619 01:05:02.158778 22357 leader_election.cc:302] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 669afdb3d91d45dd9716f3fcb1a85cf9; no voters:
I20260619 01:05:02.159533 22357 leader_election.cc:288] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:05:02.159654 22367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:02.160415 22367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [term 1 LEADER]: Becoming Leader. State: Replica: 669afdb3d91d45dd9716f3fcb1a85cf9, State: Running, Role: LEADER
I20260619 01:05:02.160996 22367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } }
I20260619 01:05:02.161744 22357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:02.164181 22368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } } }
I20260619 01:05:02.164690 22368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:02.167238 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 669afdb3d91d45dd9716f3fcb1a85cf9. Latest consensus state: current_term: 1 leader_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "669afdb3d91d45dd9716f3fcb1a85cf9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46059 } } }
I20260619 01:05:02.167707 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:02.168287 22374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:02.174547 22374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:02.184134 22374 catalog_manager.cc:1347] Generated new cluster ID: c5e56b87de084a478da0334e83809571
I20260619 01:05:02.184283 22374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:02.212203 22374 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:02.212641 22374 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:02.220965 22374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 669afdb3d91d45dd9716f3fcb1a85cf9: Generated new TSK 0
I20260619 01:05:02.221777 22374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:02.228041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22290
W20260619 01:05:02.229221 22352 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46587
W20260619 01:05:02.229883 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22290: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:05:02.230038 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22290
2026-06-19T01:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1850 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:02 dist-test-slave-3rxd krb5kdc[22388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:02 dist-test-slave-3rxd krb5kdc[22388](info): set up 2 sockets
Jun 19 01:05:02 dist-test-slave-3rxd krb5kdc[22388](info): commencing operation
krb5kdc: starting...
W20260619 01:05:03.383307 388 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:03 dist-test-slave-3rxd krb5kdc[22388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:03.634006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44017
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:03.955322 22401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:03.955574 22401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:03.955662 22401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:03.962569 22401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:03.962661 22401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:03.962795 22401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:03.962898 22401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:03.962981 22401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:03.972575 22401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:03.973728 22401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:03.975066 22401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:03.986338 22409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:03.986403 22406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:03.986371 22407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:03.987280 22401 server_base.cc:1034] running on GCE node
I20260619 01:05:03.988023 22401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:03.989228 22401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:03.990456 22401 hybrid_clock.cc:648] HybridClock initialized: now 1781831103990413 us; error 31 us; skew 500 ppm
I20260619 01:05:03.990890 22401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:03 dist-test-slave-3rxd krb5kdc[22388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:03.997355 22401 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:03.998890 22401 webserver.cc:460] Webserver started at http://127.0.97.62:38343/ using document root <none> and password file <none>
I20260619 01:05:03.999558 22401 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:03.999699 22401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:03.999993 22401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:04.002884 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "ba0da980afc44644bc6a70974df2d0a1"
format_stamp: "Formatted at 2026-06-19 01:05:03 on dist-test-slave-3rxd"
server_key: "515e258887009ab45b199295055b168c"
server_key_iv: "7e1bb9a598565bfce40e63ec7f54e1cc"
server_key_version: "encryptionkey@0"
I20260619 01:05:04.003629 22401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "ba0da980afc44644bc6a70974df2d0a1"
format_stamp: "Formatted at 2026-06-19 01:05:03 on dist-test-slave-3rxd"
server_key: "515e258887009ab45b199295055b168c"
server_key_iv: "7e1bb9a598565bfce40e63ec7f54e1cc"
server_key_version: "encryptionkey@0"
I20260619 01:05:04.008512 22401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 01:05:04.011663 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:04.012871 22401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260619 01:05:04.013053 22401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "ba0da980afc44644bc6a70974df2d0a1"
format_stamp: "Formatted at 2026-06-19 01:05:03 on dist-test-slave-3rxd"
server_key: "515e258887009ab45b199295055b168c"
server_key_iv: "7e1bb9a598565bfce40e63ec7f54e1cc"
server_key_version: "encryptionkey@0"
I20260619 01:05:04.013257 22401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:04.042832 22401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:04.046571 22401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:04.046868 22401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:04.064741 22401 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44017
I20260619 01:05:04.064757 22467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44017 every 8 connection(s)
I20260619 01:05:04.066195 22401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:04.067651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22401
I20260619 01:05:04.067813 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:04.068328 388 external_mini_cluster.cc:1428] Setting key 7b740fa2ad2ab09e7133b8bf2f713ca6
I20260619 01:05:04.070778 22468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:04.081187 22468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1: Bootstrap starting.
Jun 19 01:05:04 dist-test-slave-3rxd krb5kdc[22388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:05:04.084791 22468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:04.085803 22468 log.cc:824] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:04.089063 22468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1: No bootstrap required, opened a new log
I20260619 01:05:04.089836 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:04.070174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38813 (local address 127.0.97.62:44017)
0619 01:05:04.071016 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:04.071028 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:04.071077 (+ 49us) server_negotiation.cc:407] Connection header received
0619 01:05:04.072659 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:04.072679 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:04.073051 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:04.073470 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:04.074932 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:04.076243 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:04.077478 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:04.078157 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:04.078364 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:04.082692 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:04.082699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:04.082754 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:04.082808 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:04.085789 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:04.086629 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:04.086636 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:04.086645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:04.086892 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:04.087317 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:04.087320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:04.087322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:04.087789 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:04.088017 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:04.088455 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 01:05:04.088807 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":347,"threads_started":1}
I20260619 01:05:04.093518 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:04.098443 22468 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } }
I20260619 01:05:04.098917 22468 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:05:04.099030 22468 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba0da980afc44644bc6a70974df2d0a1, State: Initialized, Role: FOLLOWER
I20260619 01:05:04.099694 22468 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } }
I20260619 01:05:04.099988 22468 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:05:04.100139 22468 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:05:04.100310 22468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:04.103178 22468 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } }
I20260619 01:05:04.103713 22468 leader_election.cc:302] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba0da980afc44644bc6a70974df2d0a1; no voters:
I20260619 01:05:04.104341 22468 leader_election.cc:288] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:05:04.104475 22478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:04.106019 22478 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [term 1 LEADER]: Becoming Leader. State: Replica: ba0da980afc44644bc6a70974df2d0a1, State: Running, Role: LEADER
I20260619 01:05:04.106529 22468 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:04.106627 22478 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } }
I20260619 01:05:04.109426 22480 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba0da980afc44644bc6a70974df2d0a1. Latest consensus state: current_term: 1 leader_uuid: "ba0da980afc44644bc6a70974df2d0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } } }
I20260619 01:05:04.109792 22480 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:04.110328 22479 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba0da980afc44644bc6a70974df2d0a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba0da980afc44644bc6a70974df2d0a1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44017 } } }
I20260619 01:05:04.110646 22479 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:04.118201 22471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:04.098504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38995 (local address 127.0.97.62:44017)
0619 01:05:04.098753 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:04.098762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:04.099198 (+ 436us) server_negotiation.cc:407] Connection header received
0619 01:05:04.099388 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:04.099395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:04.099476 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:04.099573 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:04.101383 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:04.102132 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:04.103906 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:04.104594 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:04.104739 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:04.107993 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:04.108002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:04.108009 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:04.108056 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:04.113111 (+ 5055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:04.114136 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:04.114142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:04.114145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:04.114308 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:04.114915 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:04.114923 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:04.114929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:04.115395 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:04.116973 (+ 1578us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:04.117136 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 01:05:04.117414 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 01:05:04.118229 22484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:04.124663 22484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:04.133338 22484 catalog_manager.cc:1347] Generated new cluster ID: 5855db3737c3425f81da75d270558b43
I20260619 01:05:04.133446 22484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:04.156325 22484 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:04.158238 22484 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:04.164912 22484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba0da980afc44644bc6a70974df2d0a1: Generated new TSK 0
I20260619 01:05:04.165895 22484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:04.178829 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22401
I20260619 01:05:04.180140 22463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:04.269932 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22401
2026-06-19T01:05:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2039 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:04 dist-test-slave-3rxd krb5kdc[22501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:04 dist-test-slave-3rxd krb5kdc[22501](info): set up 2 sockets
Jun 19 01:05:04 dist-test-slave-3rxd krb5kdc[22501](info): commencing operation
krb5kdc: starting...
W20260619 01:05:05.371135 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:05 dist-test-slave-3rxd krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:05.613451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38199
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:05.936106 22514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:05.936401 22514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:05.936501 22514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:05.943519 22514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:05.943619 22514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:05.943665 22514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:05.943723 22514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:05.943759 22514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:05.953016 22514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:05.954145 22514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:05.955427 22514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:05.966847 22519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:05.966938 22520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:05.966944 22522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:05.967718 22514 server_base.cc:1034] running on GCE node
I20260619 01:05:05.968389 22514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:05.969691 22514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:05.970903 22514 hybrid_clock.cc:648] HybridClock initialized: now 1781831105970872 us; error 44 us; skew 500 ppm
I20260619 01:05:05.971221 22514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:05 dist-test-slave-3rxd krb5kdc[22501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:05.974148 22514 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:05.975562 22514 webserver.cc:460] Webserver started at http://127.0.97.62:41231/ using document root <none> and password file <none>
I20260619 01:05:05.976178 22514 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:05.976317 22514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:05.976603 22514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:05.979457 22514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "da9f3857838346ffbf0ab96e70916ba1"
format_stamp: "Formatted at 2026-06-19 01:05:05 on dist-test-slave-3rxd"
server_key: "d92ee8be8b1eb6151d3946f1b54afa97"
server_key_iv: "9f99c70b656972c8c7915713088a48bd"
server_key_version: "encryptionkey@0"
I20260619 01:05:05.980170 22514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "da9f3857838346ffbf0ab96e70916ba1"
format_stamp: "Formatted at 2026-06-19 01:05:05 on dist-test-slave-3rxd"
server_key: "d92ee8be8b1eb6151d3946f1b54afa97"
server_key_iv: "9f99c70b656972c8c7915713088a48bd"
server_key_version: "encryptionkey@0"
I20260619 01:05:05.984853 22514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 01:05:05.988090 22529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:05.989326 22514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 01:05:05.989526 22514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "da9f3857838346ffbf0ab96e70916ba1"
format_stamp: "Formatted at 2026-06-19 01:05:05 on dist-test-slave-3rxd"
server_key: "d92ee8be8b1eb6151d3946f1b54afa97"
server_key_iv: "9f99c70b656972c8c7915713088a48bd"
server_key_version: "encryptionkey@0"
I20260619 01:05:05.989722 22514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:06.009524 22514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:06.013705 22514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:06.013973 22514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:06.031800 22514 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38199
I20260619 01:05:06.031831 22580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38199 every 8 connection(s)
I20260619 01:05:06.033075 22514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:06.035688 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22514
I20260619 01:05:06.035857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:06.036373 388 external_mini_cluster.cc:1428] Setting key f304c294a1349c3f37136cdb9f60d0bd
I20260619 01:05:06.037084 22581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:06.049142 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:06.038084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59967 (local address 127.0.97.62:38199)
0619 01:05:06.038840 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:06.038851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:06.038888 (+ 37us) server_negotiation.cc:407] Connection header received
0619 01:05:06.040374 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:06.040393 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:06.040779 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:06.041142 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:06.041702 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:06.043065 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:06.044857 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:06.045460 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:06.045577 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:06.046826 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:06.046836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:06.046849 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:06.047411 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:06.047488 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:06.047515 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:06.047603 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:06.047874 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 01:05:06.048226 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260619 01:05:06.050243 22581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1: Bootstrap starting.
I20260619 01:05:06.052731 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:06.053553 22581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:06.054761 22581 log.cc:824] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:06.057495 22581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1: No bootstrap required, opened a new log
I20260619 01:05:06.064306 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:06.057750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42853 (local address 127.0.97.62:38199)
0619 01:05:06.057962 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:06.057975 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:06.058384 (+ 409us) server_negotiation.cc:407] Connection header received
0619 01:05:06.058570 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:06.058574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:06.058631 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:06.058753 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:06.059356 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:06.060087 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:06.061990 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:06.062413 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:06.062505 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:06.063584 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:06.063588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:06.063591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:06.063610 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:06.063644 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:06.063661 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:06.063731 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:06.063872 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 01:05:06.063954 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:05:06.066910 22581 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } }
I20260619 01:05:06.067306 22581 raft_consensus.cc:374] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:05:06.067418 22581 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9f3857838346ffbf0ab96e70916ba1, State: Initialized, Role: FOLLOWER
I20260619 01:05:06.068040 22581 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } }
I20260619 01:05:06.068274 22581 raft_consensus.cc:388] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:05:06.068418 22581 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:05:06.068593 22581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:06.070772 22581 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } }
I20260619 01:05:06.071276 22581 leader_election.cc:302] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da9f3857838346ffbf0ab96e70916ba1; no voters:
I20260619 01:05:06.071892 22581 leader_election.cc:288] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:05:06.072029 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:06.072716 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [term 1 LEADER]: Becoming Leader. State: Replica: da9f3857838346ffbf0ab96e70916ba1, State: Running, Role: LEADER
I20260619 01:05:06.073242 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } }
I20260619 01:05:06.073863 22581 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:06.076120 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9f3857838346ffbf0ab96e70916ba1. Latest consensus state: current_term: 1 leader_uuid: "da9f3857838346ffbf0ab96e70916ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } } }
I20260619 01:05:06.076190 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da9f3857838346ffbf0ab96e70916ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9f3857838346ffbf0ab96e70916ba1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38199 } } }
I20260619 01:05:06.076495 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:06.076560 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1 [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:06.078027 22598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:06.083209 22598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:06.091574 22598 catalog_manager.cc:1347] Generated new cluster ID: e24bf5e541fa4cd1939a48b199624fce
I20260619 01:05:06.091670 22598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:06.126441 22598 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:06.127797 22598 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:06.142652 22598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9f3857838346ffbf0ab96e70916ba1: Generated new TSK 0
I20260619 01:05:06.143795 22598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:06.158362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22514
W20260619 01:05:06.159579 22576 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59967
W20260619 01:05:06.160243 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22514: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:05:06.160388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22514
2026-06-19T01:05:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1888 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:06 dist-test-slave-3rxd krb5kdc[22612](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:06 dist-test-slave-3rxd krb5kdc[22612](info): set up 2 sockets
Jun 19 01:05:06 dist-test-slave-3rxd krb5kdc[22612](info): commencing operation
krb5kdc: starting...
W20260619 01:05:07.255110 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:07 dist-test-slave-3rxd krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:07.485950 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44697
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:07.803438 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:07.803757 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:07.803920 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:07.810823 22625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:07.810963 22625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:07.811069 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:07.811161 22625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:07.811228 22625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:07.820662 22625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:07.821920 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:07.823237 22625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:07.834848 22630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:07.834869 22631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:07.834839 22633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:07.835711 22625 server_base.cc:1034] running on GCE node
I20260619 01:05:07.836424 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:07.837738 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:07.838953 22625 hybrid_clock.cc:648] HybridClock initialized: now 1781831107838910 us; error 67 us; skew 500 ppm
I20260619 01:05:07.839286 22625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:07 dist-test-slave-3rxd krb5kdc[22612](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:07.842137 22625 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:07.843564 22625 webserver.cc:460] Webserver started at http://127.0.97.62:45765/ using document root <none> and password file <none>
I20260619 01:05:07.844277 22625 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:07.844419 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:07.844717 22625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:07.847654 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "b672218067ca464eabe33e8e342a15cc"
format_stamp: "Formatted at 2026-06-19 01:05:07 on dist-test-slave-3rxd"
server_key: "52af79bcf0ab430f14dfb6115d19e9ca"
server_key_iv: "d21fe9b910b9abc37e852d61875e2344"
server_key_version: "encryptionkey@0"
I20260619 01:05:07.848394 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "b672218067ca464eabe33e8e342a15cc"
format_stamp: "Formatted at 2026-06-19 01:05:07 on dist-test-slave-3rxd"
server_key: "52af79bcf0ab430f14dfb6115d19e9ca"
server_key_iv: "d21fe9b910b9abc37e852d61875e2344"
server_key_version: "encryptionkey@0"
I20260619 01:05:07.853804 22625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 01:05:07.857077 22640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:07.858305 22625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 01:05:07.858480 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "b672218067ca464eabe33e8e342a15cc"
format_stamp: "Formatted at 2026-06-19 01:05:07 on dist-test-slave-3rxd"
server_key: "52af79bcf0ab430f14dfb6115d19e9ca"
server_key_iv: "d21fe9b910b9abc37e852d61875e2344"
server_key_version: "encryptionkey@0"
I20260619 01:05:07.858671 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:07.864060 22625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:07.867926 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:07.868219 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:07.885236 22625 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44697
I20260619 01:05:07.885279 22691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44697 every 8 connection(s)
I20260619 01:05:07.886529 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:07.890084 22692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:07.896188 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22625
I20260619 01:05:07.896353 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:07.896786 388 external_mini_cluster.cc:1428] Setting key 78855396da8169253ef59c3b7733c3e0
I20260619 01:05:07.900602 22692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc: Bootstrap starting.
I20260619 01:05:07.904379 22692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:07.905280 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:07.898493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38217 (local address 127.0.97.62:44697)
0619 01:05:07.899315 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:07.899330 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:07.899375 (+ 45us) server_negotiation.cc:407] Connection header received
0619 01:05:07.901665 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:07.901688 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:07.902096 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:07.902519 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:07.903463 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:07.903471 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:07.903488 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:07.903507 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:07.903576 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:07.903618 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:07.903789 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:07.903905 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 01:05:07.904093 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":348,"threads_started":1}
I20260619 01:05:07.905706 22692 log.cc:824] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:07.908782 22692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc: No bootstrap required, opened a new log
I20260619 01:05:07.908963 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:07.915303 22695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:07.912966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47813 (local address 127.0.97.62:44697)
0619 01:05:07.913154 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:07.913160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:07.913481 (+ 321us) server_negotiation.cc:407] Connection header received
0619 01:05:07.913661 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:07.913666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:07.913716 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:07.913816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:07.914658 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:07.914663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:07.914670 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 01:05:07.914686 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:07.914769 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 01:05:07.914791 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:07.914924 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:07.915001 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 01:05:07.915065 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 01:05:07.916821 22692 raft_consensus.cc:348] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } }
I20260619 01:05:07.917141 22692 raft_consensus.cc:374] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 01:05:07.917251 22692 raft_consensus.cc:729] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b672218067ca464eabe33e8e342a15cc, State: Initialized, Role: FOLLOWER
I20260619 01:05:07.917776 22692 consensus_queue.cc:260] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } }
I20260619 01:05:07.918012 22692 raft_consensus.cc:388] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 01:05:07.918152 22692 raft_consensus.cc:482] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 01:05:07.918318 22692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:07.920789 22692 raft_consensus.cc:504] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } }
I20260619 01:05:07.921262 22692 leader_election.cc:302] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b672218067ca464eabe33e8e342a15cc; no voters:
I20260619 01:05:07.921808 22692 leader_election.cc:288] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 01:05:07.921898 22702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:07.922588 22702 raft_consensus.cc:686] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [term 1 LEADER]: Becoming Leader. State: Replica: b672218067ca464eabe33e8e342a15cc, State: Running, Role: LEADER
I20260619 01:05:07.923115 22702 consensus_queue.cc:237] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } }
I20260619 01:05:07.923738 22692 sys_catalog.cc:564] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:07.925518 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b672218067ca464eabe33e8e342a15cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } } }
I20260619 01:05:07.925863 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:07.927440 22704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b672218067ca464eabe33e8e342a15cc. Latest consensus state: current_term: 1 leader_uuid: "b672218067ca464eabe33e8e342a15cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b672218067ca464eabe33e8e342a15cc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44697 } } }
I20260619 01:05:07.927795 22704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:07.928226 22708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:07.933753 22708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:07.941663 22708 catalog_manager.cc:1347] Generated new cluster ID: da0ec46b49c24f51bb8326dfd03dac80
I20260619 01:05:07.941759 22708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:07.959371 22708 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:07.959811 22708 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:07.966085 22708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b672218067ca464eabe33e8e342a15cc: Generated new TSK 0
I20260619 01:05:07.966945 22708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:07.972196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22625
W20260619 01:05:07.973203 22687 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38217
W20260619 01:05:07.973798 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22625: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 01:05:07.973930 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22625
2026-06-19T01:05:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1812 ms)
[----------] 18 tests from AuthTokenIssuingTest (33972 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:08 dist-test-slave-3rxd krb5kdc[22723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:08 dist-test-slave-3rxd krb5kdc[22723](info): set up 2 sockets
Jun 19 01:05:08 dist-test-slave-3rxd krb5kdc[22723](info): commencing operation
krb5kdc: starting...
W20260619 01:05:09.043380 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:09 dist-test-slave-3rxd krb5kdc[22723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:09.278689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:09.600267 22736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:09.600805 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:09.600986 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:09.601223 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:09.614976 22736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:09.615105 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:09.615221 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:09.615310 22736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:09.615417 22736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:09.626278 22736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:09.627453 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:09.628643 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:09.640323 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:09.640354 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:09.640411 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:09.641057 22736 server_base.cc:1034] running on GCE node
I20260619 01:05:09.641748 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:09.643033 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:09.644287 22736 hybrid_clock.cc:648] HybridClock initialized: now 1781831109644211 us; error 43 us; skew 500 ppm
I20260619 01:05:09.644587 22736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:09 dist-test-slave-3rxd krb5kdc[22723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:09.647512 22736 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:09.648736 22736 webserver.cc:460] Webserver started at http://127.0.97.62:43775/ using document root <none> and password file <none>
I20260619 01:05:09.649307 22736 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:09.649385 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:09.649639 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:09.652387 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "2bb895d15dc141598c8770f840a18a0b"
format_stamp: "Formatted at 2026-06-19 01:05:09 on dist-test-slave-3rxd"
server_key: "59a51b31721337ddeb597e877e9bc7ff"
server_key_iv: "22c59465d5f680a3145dea8e10633794"
server_key_version: "encryptionkey@0"
I20260619 01:05:09.653081 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "2bb895d15dc141598c8770f840a18a0b"
format_stamp: "Formatted at 2026-06-19 01:05:09 on dist-test-slave-3rxd"
server_key: "59a51b31721337ddeb597e877e9bc7ff"
server_key_iv: "22c59465d5f680a3145dea8e10633794"
server_key_version: "encryptionkey@0"
I20260619 01:05:09.657729 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 01:05:09.660759 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:09.661979 22736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 01:05:09.662132 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "2bb895d15dc141598c8770f840a18a0b"
format_stamp: "Formatted at 2026-06-19 01:05:09 on dist-test-slave-3rxd"
server_key: "59a51b31721337ddeb597e877e9bc7ff"
server_key_iv: "22c59465d5f680a3145dea8e10633794"
server_key_version: "encryptionkey@0"
I20260619 01:05:09.662333 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:09.674816 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:09.678792 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:09.679037 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:09.695874 22736 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33611
I20260619 01:05:09.695894 22802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33611 every 8 connection(s)
I20260619 01:05:09.697073 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:09.698450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22736
I20260619 01:05:09.698668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:09.699290 388 external_mini_cluster.cc:1428] Setting key 738f311b58391df7c17354ad54b1edd5
I20260619 01:05:09.700500 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:09.706674 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:09 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 01:05:09.731956 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.711479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54774 (local address 127.0.97.62:33611)
0619 01:05:09.712140 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:09.712145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:09.712193 (+ 48us) server_negotiation.cc:407] Connection header received
0619 01:05:09.714360 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:09.714366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:09.714587 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:09.714820 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:09.716411 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:09.717689 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:09.720007 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:09.720547 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:09.720683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:09.724752 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:09.724759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:09.724770 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:09.724803 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:09.727744 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:09.728648 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:09.728654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:09.728656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:09.728789 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:09.729265 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:09.729268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:09.729270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:09.729724 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:09.729882 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:09.730569 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 01:05:09.730923 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":331,"threads_started":1}
I20260619 01:05:09.732002 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.710328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.0.1:54774)
0619 01:05:09.711200 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.711252 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:09.713974 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:09.714855 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:09.714864 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:09.715648 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:09.716162 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:09.716188 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:09.717851 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:09.717859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:09.719821 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:09.719840 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:09.720804 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:09.720814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:09.721085 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:09.721802 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:09.721833 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:09.724526 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:09.727880 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:09.727894 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:09.727910 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:09.728520 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:09.728902 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:09.728909 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:09.728916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:09.729131 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:09.729877 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:09.729888 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:09.730128 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 01:05:09.730400 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 01:05:09.731165 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":81,"thread_start_us":353,"threads_started":1}
I20260619 01:05:09.735265 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:09.736374 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.736007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57502)
0619 01:05:09.736169 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.736272 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 01:05:09.736692 22755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
I20260619 01:05:09.738296 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.737943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57516)
0619 01:05:09.738092 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.738176 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:05:09.738590 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40459: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:05:09.755833 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40459
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:05:09.787941 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } attempt: 1
I20260619 01:05:09.789171 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.788733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57520)
0619 01:05:09.788926 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.789047 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 01:05:09.790110 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.789768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57532)
0619 01:05:09.789909 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.789980 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:05:09.790294 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40459: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
I20260619 01:05:09.858529 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } attempt: 2
I20260619 01:05:09.859732 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.859255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57536)
0619 01:05:09.859503 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.859598 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 01:05:09.860729 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.860372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57548)
0619 01:05:09.860528 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.860603 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:05:09.860914 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40459: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
I20260619 01:05:09.952169 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } attempt: 3
I20260619 01:05:09.953470 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.953010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57562)
0619 01:05:09.953219 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.953317 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 01:05:09.954487 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:09.954141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57576)
0619 01:05:09.954283 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:09.954355 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 01:05:09.954675 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40459: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
W20260619 01:05:10.074898 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:10.075259 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:10.075382 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:10.075518 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:10.082645 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:10.082820 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:10.082937 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:10.083021 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:10.083127 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:10.092856 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:10.094125 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:10.095541 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:05:10.098006 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } attempt: 4
I20260619 01:05:10.099148 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.098622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57590)
0619 01:05:10.098937 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.099024 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260619 01:05:10.100095 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.099743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57596)
0619 01:05:10.099900 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.099975 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 01:05:10.100364 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40459: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40459: connect: Connection refused (error 111)
W20260619 01:05:10.107295 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:10.107357 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:10.107635 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:10.108794 22810 server_base.cc:1034] running on GCE node
I20260619 01:05:10.109546 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:10.110996 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:10.112227 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781831110112168 us; error 30 us; skew 500 ppm
I20260619 01:05:10.112560 22810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:10.115653 22810 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:05:10.117198 22810 webserver.cc:460] Webserver started at http://127.0.97.61:45941/ using document root <none> and password file <none>
I20260619 01:05:10.117877 22810 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:10.118005 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:10.118314 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:10.121235 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data/instance:
uuid: "b9f2e0c6839a48648a920a9354fbd2c8"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "6ef99a4d7e269c7956ecf58993180c94"
server_key_iv: "7c0201e319f1ac0e6f79bc7670367bd5"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.122021 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal/instance:
uuid: "b9f2e0c6839a48648a920a9354fbd2c8"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "6ef99a4d7e269c7956ecf58993180c94"
server_key_iv: "7c0201e319f1ac0e6f79bc7670367bd5"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.126963 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 01:05:10.130364 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:10.132319 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:05:10.132524 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "b9f2e0c6839a48648a920a9354fbd2c8"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "6ef99a4d7e269c7956ecf58993180c94"
server_key_iv: "7c0201e319f1ac0e6f79bc7670367bd5"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.132763 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:10.145675 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:10.149497 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:10.149751 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:10.168298 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40459
I20260619 01:05:10.168315 22876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40459 every 8 connection(s)
I20260619 01:05:10.169569 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:05:10.173080 22877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:10.177809 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22810
I20260619 01:05:10.177992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-1/wal/instance
I20260619 01:05:10.178476 388 external_mini_cluster.cc:1428] Setting key 44d3b067540cb6537cc6dfa3b93226be
I20260619 01:05:10.179865 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:05:10.203712 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.183420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.0.97.62:33611)
0619 01:05:10.183623 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.183631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.185209 (+ 1578us) server_negotiation.cc:407] Connection header received
0619 01:05:10.188090 (+ 2881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.188094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.188164 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.188254 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.190053 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.190929 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.193534 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.193959 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.194045 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.198195 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.198200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.198203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.198234 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.200835 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.201521 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.201526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.201529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.201648 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.202042 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.202048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.202051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.202322 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.202473 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.203303 (+ 830us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.203485 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:05:10.205180 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.183883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.0.1:54780)
0619 01:05:10.184906 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.184958 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.187810 (+ 2852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.188509 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.188522 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.189312 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.189856 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.189873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.191095 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.191104 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.193331 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.193351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.194194 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.194203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.194554 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.195312 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.195347 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.197979 (+ 2632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.200973 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.200982 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.200997 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.201410 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.201750 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.201754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.201758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.201935 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.202451 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.202459 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.202906 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.203232 (+ 326us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.203789 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":496,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260619 01:05:10.206979 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:10.220832 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.207766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:57600)
0619 01:05:10.207927 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.207941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.208013 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.208961 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.208965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.209360 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.209470 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.209477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.210860 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.210869 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.211684 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.211694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.212518 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.212523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.212698 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.213224 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.213239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.215142 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.218448 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.218455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.218459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.218878 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.219224 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.219230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.219251 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.219379 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.219916 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.219921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.220049 (+ 128us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.220194 (+ 145us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.220444 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 01:05:10.220933 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.207982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.97.61:40459)
0619 01:05:10.208560 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.208567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.208591 (+ 24us) server_negotiation.cc:407] Connection header received
0619 01:05:10.208631 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.208637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.208785 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.208986 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.209648 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.210666 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.211846 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.212379 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.212512 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.215343 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.215352 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.215363 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.215388 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.218321 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.218993 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.219000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.219004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.219115 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.219493 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.219498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.219502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.219797 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.219997 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.220354 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.220553 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":292,"threads_started":1}
I20260619 01:05:10.223323 22877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:10.224519 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.223921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37550)
0619 01:05:10.224127 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.224289 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 01:05:10.224766 22829 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.226315 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.225897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37552)
0619 01:05:10.226073 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.226172 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:05:10.226543 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 01:05:10.237859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34575
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:05:10.275910 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 1
I20260619 01:05:10.277190 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.276647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37566)
0619 01:05:10.276867 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.276995 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:05:10.278157 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.277755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37578)
0619 01:05:10.277923 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.278024 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 01:05:10.278364 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.346601 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 2
I20260619 01:05:10.347909 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.347322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37592)
0619 01:05:10.347567 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.347710 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 01:05:10.348987 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.348515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:37602)
0619 01:05:10.348706 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.348794 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:05:10.349220 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.399698 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } attempt: 5
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:10.412290 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.400605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.0.97.61:40459)
0619 01:05:10.400773 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.400781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.400821 (+ 40us) server_negotiation.cc:407] Connection header received
0619 01:05:10.401035 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.401043 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.401142 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.401248 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.402045 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.402683 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.403797 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.404201 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.404330 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.407327 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.407331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.407334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.407361 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.409778 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.410448 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.410452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.410454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.410575 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.410972 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.410979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.410982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.411239 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.411392 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.411751 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.411956 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:05:10.412288 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.400512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:51008)
0619 01:05:10.400737 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.400758 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.400898 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.401282 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.401287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.401723 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.401905 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.401913 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.402849 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.402854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.403653 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.403662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.404358 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.404363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.404554 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.405217 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.405233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.407152 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.409932 (+ 2780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.409937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.409942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.410307 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.410697 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.410702 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.410725 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.410866 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.411356 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.411361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.411514 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.411652 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.411930 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 01:05:10.413708 22803 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:10.414661 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.414301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34036)
0619 01:05:10.414451 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.414526 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 01:05:10.415690 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.415366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34044)
0619 01:05:10.415518 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.415596 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 01:05:10.415896 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.440469 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 3
I20260619 01:05:10.441733 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.441158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34058)
0619 01:05:10.441409 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.441527 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 01:05:10.442945 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.442403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34060)
0619 01:05:10.442648 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.442786 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260619 01:05:10.443168 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.467101 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 1
I20260619 01:05:10.468238 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.467661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34064)
0619 01:05:10.467881 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.468043 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 01:05:10.469388 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.468951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34080)
0619 01:05:10.469126 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.469231 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:05:10.469686 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.537894 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 2
I20260619 01:05:10.539144 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.538495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34092)
0619 01:05:10.538784 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.538913 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 01:05:10.540323 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.539887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34098)
0619 01:05:10.540067 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.540170 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 01:05:10.540582 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
W20260619 01:05:10.577075 22884 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:10.577344 22884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:10.577433 22884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:10.577584 22884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:10.584597 22884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:10.584672 22884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:10.584726 22884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:10.584796 22884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:10.584903 22884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:10.586391 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 4
I20260619 01:05:10.587610 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.587043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34112)
0619 01:05:10.587310 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.587419 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:05:10.588616 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.588263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34120)
0619 01:05:10.588416 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.588496 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:05:10.588869 22877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.595055 22884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33611,127.0.97.61:40459,127.0.97.60:34575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:34575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:10.596244 22884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:10.597450 22884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:10.608441 22889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:10.608500 22892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:10.608407 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:10.609612 22884 server_base.cc:1034] running on GCE node
I20260619 01:05:10.610361 22884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:10.611701 22884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:10.612919 22884 hybrid_clock.cc:648] HybridClock initialized: now 1781831110612866 us; error 24 us; skew 500 ppm
I20260619 01:05:10.613189 22884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:10.616221 22884 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:05:10.617465 22884 webserver.cc:460] Webserver started at http://127.0.97.60:33341/ using document root <none> and password file <none>
I20260619 01:05:10.618055 22884 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:10.618139 22884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:10.618383 22884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:10.621165 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data/instance:
uuid: "7ed16d75904e47f19abbed5f5498ffc1"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "398d57f9b3b1f2c1cf018e507fae657b"
server_key_iv: "319cdeb1d1718abe8a4f4ba7cc681e40"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.621866 22884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal/instance:
uuid: "7ed16d75904e47f19abbed5f5498ffc1"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "398d57f9b3b1f2c1cf018e507fae657b"
server_key_iv: "319cdeb1d1718abe8a4f4ba7cc681e40"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.626493 22884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 01:05:10.629765 22899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:10.631055 22884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 01:05:10.631212 22884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "7ed16d75904e47f19abbed5f5498ffc1"
format_stamp: "Formatted at 2026-06-19 01:05:10 on dist-test-slave-3rxd"
server_key: "398d57f9b3b1f2c1cf018e507fae657b"
server_key_iv: "319cdeb1d1718abe8a4f4ba7cc681e40"
server_key_version: "encryptionkey@0"
I20260619 01:05:10.631417 22884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:10.646785 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 3
I20260619 01:05:10.647928 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.647395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34130)
0619 01:05:10.647641 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.647739 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 01:05:10.648952 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.648574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34136)
0619 01:05:10.648743 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.648827 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:05:10.649168 22803 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:34575: Network error: Client connection negotiation failed: client connection to 127.0.97.60:34575: connect: Connection refused (error 111)
I20260619 01:05:10.652264 22884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:10.656023 22884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:10.656277 22884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:10.673976 22884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:34575
I20260619 01:05:10.674007 22950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:34575 every 8 connection(s)
I20260619 01:05:10.675288 22884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:05:10.678845 22951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:10.680711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 22884
I20260619 01:05:10.680886 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781830760188020-388-0/minicluster-data/master-2/wal/instance
I20260619 01:05:10.681474 388 external_mini_cluster.cc:1428] Setting key 13a77dd3999bd8ebe52ba47a55844f51
I20260619 01:05:10.688231 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:05:10.707571 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.690889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.0.97.62:33611)
0619 01:05:10.691112 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.691118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.691763 (+ 645us) server_negotiation.cc:407] Connection header received
0619 01:05:10.691999 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.692003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.692063 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.692166 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.693163 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.693862 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.695503 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.696068 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.696197 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.700300 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.700305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.700308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.700342 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.704591 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.705466 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.705472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.705475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.705654 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.706161 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.706167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.706170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.706503 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.706645 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.707173 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.707333 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:05:10.707948 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.690798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33611 (local address 127.0.0.1:54544)
0619 01:05:10.691610 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.691659 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.691799 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.692224 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.692230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.692782 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.692994 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.693002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.694004 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.694008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.695329 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.695340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.696263 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.696270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.696516 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.697358 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.697383 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.700103 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.704725 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.704739 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.704746 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.705289 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.705772 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.705775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.705797 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.705974 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.706651 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.706657 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.706884 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.707068 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.707371 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":433,"threads_started":1}
I20260619 01:05:10.707966 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.683342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.0.97.60:34575)
0619 01:05:10.684085 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.684095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.684145 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:05:10.685598 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.685621 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.685978 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.686266 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.687885 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.689125 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.690895 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.691750 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.691940 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.696153 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.696165 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.696178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.696220 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.699660 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.700530 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.700540 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.700552 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.700736 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.701181 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.701187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.701191 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.705355 (+ 4164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.705613 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.706103 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.706514 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"spinlock_wait_cycles":21376,"thread_start_us":338,"threads_started":1}
I20260619 01:05:10.709609 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:05:10.724840 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.710442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40459 (local address 127.0.0.1:51014)
0619 01:05:10.710743 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.710764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.710861 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.711220 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.711224 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.711683 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.711860 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.711872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.713291 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.713295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.714646 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.714657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.715581 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.715585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.715705 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.716296 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.716320 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.718881 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.722428 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.722432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.722435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.722863 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.723242 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.723247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.723251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.723401 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.723926 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.723931 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.724071 (+ 140us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.724202 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.724469 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 01:05:10.724865 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.710400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.0.97.61:40459)
0619 01:05:10.710591 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.710599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.710822 (+ 223us) server_negotiation.cc:407] Connection header received
0619 01:05:10.710971 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.710978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.711060 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.711176 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.712013 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.713131 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.714798 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.715432 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.715558 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.719055 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.719060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.719069 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.719100 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.722304 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.722992 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.722997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.722999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.723130 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.723518 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.723527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.723529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.723779 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.723947 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.724300 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.724500 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 01:05:10.725891 22951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:10.727252 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.711413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.0.97.62:33611)
0619 01:05:10.711596 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.711601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.711621 (+ 20us) server_negotiation.cc:407] Connection header received
0619 01:05:10.711668 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.711673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.711738 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.711840 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.713286 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.714377 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.715947 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.716578 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.716713 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.721331 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.721339 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.721343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.721389 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.724177 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.724786 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.724792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.724795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.725144 (+ 349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.725542 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.725547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.725550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.725915 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.726174 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.726480 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.726699 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:10.739493 22955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.726588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34164)
0619 01:05:10.726789 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.726810 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.726907 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.727243 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.727246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.727656 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.727776 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.727784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.728957 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.728961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.729914 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.729924 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.730617 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.730622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.730855 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.731384 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.731400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.733348 (+ 1948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.736933 (+ 3585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.736938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.736942 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.737506 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.737856 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.737861 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.737864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.737985 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.738454 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.738458 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.738594 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.738767 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.739109 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 01:05:10.739534 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.726841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.0.97.60:34575)
0619 01:05:10.726995 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.727000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.727016 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:05:10.727059 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.727064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.727128 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.727218 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.727906 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.728797 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.730070 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.730461 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.730592 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.733533 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.733538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.733542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.733577 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.736829 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.737622 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.737626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.737628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.737738 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.738097 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.738101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.738106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.738345 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.738489 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.738840 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.739251 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":54}
I20260619 01:05:10.743706 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.728806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.0.97.61:40459)
0619 01:05:10.728981 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.728986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.729002 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:05:10.729044 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.729049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.729121 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.729212 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.730281 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.731060 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.732224 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.732606 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.732734 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.737857 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.737862 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.737867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.737904 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.741158 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.741913 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.741917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.741919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.742053 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.742479 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.742482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.742484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.742776 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.742997 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.743305 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.743463 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:05:10.746347 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: Bootstrap starting.
I20260619 01:05:10.748574 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:10.750406 22951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:10.751808 22951 log.cc:824] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:10.755012 22951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: No bootstrap required, opened a new log
I20260619 01:05:10.752081 22778 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2bb895d15dc141598c8770f840a18a0b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54556
I20260619 01:05:10.765717 22951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:10.766134 22951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed16d75904e47f19abbed5f5498ffc1, State: Initialized, Role: FOLLOWER
I20260619 01:05:10.766650 22951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:10.767535 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.767874 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:10.768879 22951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 01:05:10.778815 22968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:05:10.778987 22968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:05:10.776492 22884 master_runner.cc:186] Error getting master registration for 127.0.97.62:33611: OK, instance_id { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" instance_seqno: 1781831109691010 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:05:10.787752 22884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:05:10.826453 22803 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 4
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:05:10.844260 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.827474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.0.97.60:34575)
0619 01:05:10.827697 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.827703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.827794 (+ 91us) server_negotiation.cc:407] Connection header received
0619 01:05:10.827995 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.828001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.828090 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.828218 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.829189 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.830425 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.832293 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.833133 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.833278 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.837647 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.837657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.837664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.837716 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.841186 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.841958 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.841967 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.841974 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.842114 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.842545 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.842552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.842556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.842992 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.843163 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.843595 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.843813 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:05:10.844260 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.827372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34178)
0619 01:05:10.827685 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.827706 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.827846 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.828278 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.828284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.828809 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.829029 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.829042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.830612 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.830619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.832134 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.832149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.833310 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.833316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.833517 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.834354 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.834378 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.837417 (+ 3039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.841357 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.841363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.841368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.841822 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.842248 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.842254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.842258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.842425 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.843148 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.843153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.843325 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.843489 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.843813 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260619 01:05:10.849490 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: Bootstrap starting.
I20260619 01:05:10.853025 22803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:10.854144 22803 log.cc:824] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:10.857132 22803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: No bootstrap required, opened a new log
I20260619 01:05:10.862506 22803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:10.862977 22803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb895d15dc141598c8770f840a18a0b, State: Initialized, Role: FOLLOWER
I20260619 01:05:10.863459 22803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:10.864524 22971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.864923 22971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:10.865861 22803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:10.866031 22778 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 01:05:10.866269 22778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:10.869856 22778 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:10.873576 22778 leader_election.cc:288] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [CANDIDATE]: Term 1 election: Requested vote from peers b9f2e0c6839a48648a920a9354fbd2c8 (127.0.97.61:40459), 7ed16d75904e47f19abbed5f5498ffc1 (127.0.97.60:34575)
I20260619 01:05:10.876044 22926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bb895d15dc141598c8770f840a18a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7ed16d75904e47f19abbed5f5498ffc1"
I20260619 01:05:10.876505 22926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:10.880228 22926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb895d15dc141598c8770f840a18a0b in term 1.
I20260619 01:05:10.883003 22755 leader_election.cc:302] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2bb895d15dc141598c8770f840a18a0b, 7ed16d75904e47f19abbed5f5498ffc1; no voters:
I20260619 01:05:10.883533 22971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:10.884292 22971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [term 1 LEADER]: Becoming Leader. State: Replica: 2bb895d15dc141598c8770f840a18a0b, State: Running, Role: LEADER
I20260619 01:05:10.884905 22971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
W20260619 01:05:10.886770 22982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:05:10.886893 22982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:05:10.888113 22877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } attempt: 5
I20260619 01:05:10.878008 22736 master_runner.cc:186] Error getting master registration for 127.0.97.61:40459: OK, instance_id { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" instance_seqno: 1781831110163127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:05:10.897159 22736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 01:05:10 dist-test-slave-3rxd krb5kdc[22723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260619 01:05:10.909071 22905 tablet.cc:2369] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:05:10.909231 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb895d15dc141598c8770f840a18a0b. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.909757 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:10.915970 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.888873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:34575 (local address 127.0.0.1:34182)
0619 01:05:10.889251 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:10.889272 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:10.889392 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:10.890352 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:10.890358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:10.890955 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:10.891182 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.891199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.892795 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.892802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.894298 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:10.894315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.895628 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.895635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:10.895870 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.896622 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:10.896654 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:10.900387 (+ 3733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:10.906202 (+ 5815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.906211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.906218 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.906917 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.907525 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:10.907534 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:10.907540 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:10.907773 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:10.914414 (+ 6641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:10.914423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:10.914652 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 01:05:10.914881 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 01:05:10.915313 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260619 01:05:10.918314 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.889627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.0.97.60:34575)
0619 01:05:10.889814 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.889823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.889844 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:05:10.889906 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.889921 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.890035 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.890144 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.891404 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.892608 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.894513 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.895286 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.895416 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:10.900685 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:10.900696 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:10.900705 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:10.900765 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:10.905950 (+ 5185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.907120 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.907130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.907136 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.907349 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:10.913578 (+ 6229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:10.913593 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:10.913600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:10.914120 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:10.914326 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:10.917325 (+ 2999us) server_negotiation.cc:299] Negotiation successful
0619 01:05:10.917669 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:05:10.923298 22990 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:10.924333 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: Bootstrap starting.
I20260619 01:05:10.925360 22990 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:10.930063 22877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: Neither blocks nor log segments found. Creating new log.
W20260619 01:05:10.931825 22757 tablet.cc:2369] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:05:10.937757 22877 log.cc:824] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:10.947382 22926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [term 1 FOLLOWER]: Refusing update from remote peer 2bb895d15dc141598c8770f840a18a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:05:10.948359 22985 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:05:10.959422 22877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: No bootstrap required, opened a new log
I20260619 01:05:10.968986 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb895d15dc141598c8770f840a18a0b. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.969298 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:10.978843 22995 mvcc.cc:205] Tried to move back new op lower bound from 7298380230429437952 to 7298380230190067712. Current Snapshot: MvccSnapshot[applied={T|T < 7298380230429437952}]
I20260619 01:05:10.985529 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.985819 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:10.989648 22990 catalog_manager.cc:1347] Generated new cluster ID: 7b259b4147bf4e0fac2677daf13f72f0
I20260619 01:05:10.989753 22990 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:10.996721 22957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:10.997059 22957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:11.015751 22807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.940073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.0.97.62:33611)
0619 01:05:10.940281 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.940290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.946674 (+ 6384us) server_negotiation.cc:407] Connection header received
0619 01:05:10.951016 (+ 4342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.951022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.951116 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.951189 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.962859 (+ 11670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.964173 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.970828 (+ 6655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.971544 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:10.971656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.006966 (+ 35310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:11.006974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:11.006980 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:11.007034 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:11.011138 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.012359 (+ 1221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.012365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.012369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.012543 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.013270 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.013276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.013279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.013636 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:11.013895 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.014750 (+ 855us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.015143 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:05:11.015589 22877 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:11.015981 22877 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f2e0c6839a48648a920a9354fbd2c8, State: Initialized, Role: FOLLOWER
I20260619 01:05:11.016556 22877 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } }
I20260619 01:05:11.017895 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:11.018575 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:11.025214 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.939267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.0.97.61:40459)
0619 01:05:10.939906 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.939914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:10.967115 (+ 27201us) server_negotiation.cc:407] Connection header received
0619 01:05:10.969712 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:10.969719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:10.969796 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:10.969865 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:10.976782 (+ 6917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:10.978090 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.006955 (+ 28865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.007681 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.008162 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.017383 (+ 9221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:11.017389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:11.017394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:11.017437 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:11.021282 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.022416 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.022423 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.022426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.022583 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.023222 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.023228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.023231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.023581 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:11.023813 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.024603 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.024845 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
I20260619 01:05:11.032583 22877 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:11.032686 22852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2bb895d15dc141598c8770f840a18a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9f2e0c6839a48648a920a9354fbd2c8"
I20260619 01:05:11.033022 22851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:11.037009 22851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [term 1 FOLLOWER]: Refusing update from remote peer 2bb895d15dc141598c8770f840a18a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:05:11.040316 22852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2bb895d15dc141598c8770f840a18a0b in term 1.
I20260619 01:05:11.040966 22971 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:05:11.047410 22990 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:11.049248 22990 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:11.053840 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:10.967295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.97.60:34575)
0619 01:05:10.967488 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:10.967494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.004773 (+ 37279us) server_negotiation.cc:407] Connection header received
0619 01:05:11.005369 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.005374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.005451 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.005526 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.011947 (+ 6421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.013095 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.021311 (+ 8216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.021928 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.022008 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.030491 (+ 8483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:11.030497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:11.030505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:11.030551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:11.035655 (+ 5104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.050988 (+ 15333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.050994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.050998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.051170 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:11.051643 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:11.051648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:11.051651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:11.051999 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:11.052229 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.052907 (+ 678us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.053285 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:05:11.074786 22990 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b: Generated new TSK 0
I20260619 01:05:11.075879 22990 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:11.082348 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bb895d15dc141598c8770f840a18a0b. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:11.082651 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:11.083792 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:11.084053 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bb895d15dc141598c8770f840a18a0b [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:11.084509 23007 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2bb895d15dc141598c8770f840a18a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb895d15dc141598c8770f840a18a0b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33611 } } peers { permanent_uuid: "b9f2e0c6839a48648a920a9354fbd2c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40459 } } peers { permanent_uuid: "7ed16d75904e47f19abbed5f5498ffc1" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 34575 } } }
I20260619 01:05:11.098253 23007 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:11.133725 23018 catalog_manager.cc:1259] Loaded cluster ID: 7b259b4147bf4e0fac2677daf13f72f0
I20260619 01:05:11.133893 23018 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: loading cluster ID for follower catalog manager: success
I20260619 01:05:11.137857 23018 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: acquiring CA information for follower catalog manager: success
I20260619 01:05:11.140728 23018 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9f2e0c6839a48648a920a9354fbd2c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 01:05:11.206481 22807 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.193367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.97.62:33611)
0619 01:05:11.193568 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.193575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.202801 (+ 9226us) server_negotiation.cc:407] Connection header received
0619 01:05:11.204778 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.204785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.204892 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.204978 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.206206 (+ 1228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54570: BlockingRecv error: recv got EOF from 127.0.0.1:54570 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 01:05:11.210494 22881 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.197195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.0.97.61:40459)
0619 01:05:11.197373 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.197379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.207902 (+ 10523us) server_negotiation.cc:407] Connection header received
0619 01:05:11.208868 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.208874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.208969 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.209060 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.210235 (+ 1175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51030: BlockingRecv error: recv got EOF from 127.0.0.1:51030 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:05:11.213514 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.203352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.0.97.60:34575)
0619 01:05:11.203523 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.203529 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.211295 (+ 7766us) server_negotiation.cc:407] Connection header received
0619 01:05:11.212204 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.212210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.212313 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.212389 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.213283 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34204: BlockingRecv error: recv got EOF from 127.0.0.1:34204 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:05:11.262259 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.247008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.0.97.61:40459)
0619 01:05:11.247211 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.247217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.254297 (+ 7080us) server_negotiation.cc:407] Connection header received
0619 01:05:11.254822 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.254829 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.255051 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.255197 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.255873 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.257059 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.259176 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.259849 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.259951 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.260688 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.261394 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.261512 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.261747 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.261844 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 01:05:11.304708 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.295576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.0.97.60:34575)
0619 01:05:11.295772 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.295779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.302588 (+ 6809us) server_negotiation.cc:407] Connection header received
0619 01:05:11.303191 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.303197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.303326 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.303421 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.304466 (+ 1045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34206: BlockingRecv error: recv got EOF from 127.0.0.1:34206 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 01:05:11.352348 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.338451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.97.61:40459)
0619 01:05:11.338647 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.338654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.345494 (+ 6840us) server_negotiation.cc:407] Connection header received
0619 01:05:11.345963 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.345970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.346093 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.346155 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.346788 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.347948 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.349964 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.350749 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.350842 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.351452 (+ 610us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.351604 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.351665 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.351897 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.351987 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 01:05:11.400243 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.391099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.0.97.60:34575)
0619 01:05:11.391335 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.391341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.398149 (+ 6808us) server_negotiation.cc:407] Connection header received
0619 01:05:11.398656 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.398665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.398786 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.398884 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.399968 (+ 1084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34208: BlockingRecv error: recv got EOF from 127.0.0.1:34208 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:05:11.437597 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.428584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.97.61:40459)
0619 01:05:11.428779 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.428785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.431331 (+ 2546us) server_negotiation.cc:407] Connection header received
0619 01:05:11.431534 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.431541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.431639 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.431748 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.432218 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.433443 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.435429 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.436051 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.436176 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.436626 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.436796 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.436881 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.437148 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.437227 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 01:05:11.452888 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.448342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.0.97.60:34575)
0619 01:05:11.448518 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.448524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.451274 (+ 2750us) server_negotiation.cc:407] Connection header received
0619 01:05:11.451473 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.451479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.451590 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.451709 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.452623 (+ 914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34212: BlockingRecv error: recv got EOF from 127.0.0.1:34212 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:05:11.472169 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.464802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.0.97.61:40459)
0619 01:05:11.465071 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.465078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.465628 (+ 550us) server_negotiation.cc:407] Connection header received
0619 01:05:11.465822 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.465828 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.465919 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.466044 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.466761 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.468028 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.470030 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.470742 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.470872 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.471411 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.471537 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.471633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.471784 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.471886 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
W20260619 01:05:11.480753 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.478552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.0.97.60:34575)
0619 01:05:11.478833 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.478840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.479337 (+ 497us) server_negotiation.cc:407] Connection header received
0619 01:05:11.479476 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.479483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.479600 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.479714 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.480534 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34226: BlockingRecv error: recv got EOF from 127.0.0.1:34226 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 01:05:11.499284 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.491795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.0.97.61:40459)
0619 01:05:11.491960 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.491967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.492480 (+ 513us) server_negotiation.cc:407] Connection header received
0619 01:05:11.492657 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.492663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.492761 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.492875 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.493330 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.494571 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.496555 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.497273 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.497392 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.497928 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.498084 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.498811 (+ 727us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.498870 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.498951 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 01:05:11.508198 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.506027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.97.60:34575)
0619 01:05:11.506198 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.506203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.506697 (+ 494us) server_negotiation.cc:407] Connection header received
0619 01:05:11.506888 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.506894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.507006 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.507117 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.507914 (+ 797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34242: BlockingRecv error: recv got EOF from 127.0.0.1:34242 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 01:05:11.535540 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.528334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.0.97.61:40459)
0619 01:05:11.528530 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.528537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.529068 (+ 531us) server_negotiation.cc:407] Connection header received
0619 01:05:11.529260 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.529266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.529355 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.529461 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.529932 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.531253 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.533070 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.533837 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.533957 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.534737 (+ 780us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.534892 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.534995 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.535165 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.535257 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 01:05:11.544778 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.542333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.97.60:34575)
0619 01:05:11.542525 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.542531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.543325 (+ 794us) server_negotiation.cc:407] Connection header received
0619 01:05:11.543523 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.543529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.543619 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.543738 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.544554 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34248: BlockingRecv error: recv got EOF from 127.0.0.1:34248 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 01:05:11.586779 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.580020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.97.61:40459)
0619 01:05:11.580241 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.580248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.580787 (+ 539us) server_negotiation.cc:407] Connection header received
0619 01:05:11.580992 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.580999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.581106 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.581221 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.581683 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.582796 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.584780 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.585397 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.585513 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.586018 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.586139 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.586220 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.586391 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.586461 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 01:05:11.600018 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.598010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.97.60:34575)
0619 01:05:11.598214 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.598220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.598492 (+ 272us) server_negotiation.cc:407] Connection header received
0619 01:05:11.598697 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.598705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.598845 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.598938 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.599790 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34256: BlockingRecv error: recv got EOF from 127.0.0.1:34256 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 01:05:11.673880 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.668093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.0.97.61:40459)
0619 01:05:11.668310 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.668316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.668638 (+ 322us) server_negotiation.cc:407] Connection header received
0619 01:05:11.668820 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.668838 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.669000 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.669081 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.669530 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.670417 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.672040 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.672593 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.672689 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.673133 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.673237 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.673329 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.673501 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.673574 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 01:05:11.686491 22954 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:11.684681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.97.60:34575)
0619 01:05:11.684888 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.684895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.685132 (+ 237us) server_negotiation.cc:407] Connection header received
0619 01:05:11.685258 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.685261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.685345 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.685421 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:11.686259 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34260: BlockingRecv error: recv got EOF from 127.0.0.1:34260 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:05:11.780965 22968 catalog_manager.cc:1259] Loaded cluster ID: 7b259b4147bf4e0fac2677daf13f72f0
I20260619 01:05:11.781060 22968 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: loading cluster ID for follower catalog manager: success
I20260619 01:05:11.783581 22968 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: acquiring CA information for follower catalog manager: success
I20260619 01:05:11.785161 22968 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7ed16d75904e47f19abbed5f5498ffc1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:05:11.823146 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.818263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.97.61:40459)
0619 01:05:11.818423 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.818427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.818829 (+ 402us) server_negotiation.cc:407] Connection header received
0619 01:05:11.819013 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.819018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.819088 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.819185 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.819627 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.820362 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.821620 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.822032 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.822124 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.822493 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.822629 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.822699 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.822854 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.822915 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:05:11.835758 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.828813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.0.97.60:34575)
0619 01:05:11.829091 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.829097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.829501 (+ 404us) server_negotiation.cc:407] Connection header received
0619 01:05:11.829746 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.829751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.829927 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.830091 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.830563 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.831590 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.833161 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.833613 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.833832 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.834381 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.835081 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.835253 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.835320 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.835413 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 01:05:11.852233 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.842240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.0.97.62:33611)
0619 01:05:11.845725 (+ 3485us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.845732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.846047 (+ 315us) server_negotiation.cc:407] Connection header received
0619 01:05:11.846279 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.846283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.846433 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.846599 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.847076 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.847773 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.849159 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.849800 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.849913 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.851138 (+ 1225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.851612 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.851730 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.851854 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.851933 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3355,"thread_start_us":3252,"threads_started":1}
I20260619 01:05:11.854624 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.842396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.97.61:40459)
0619 01:05:11.842632 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.842640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.848916 (+ 6276us) server_negotiation.cc:407] Connection header received
0619 01:05:11.849111 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.849116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.849198 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.849295 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.849891 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.850652 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.852781 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.853384 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.853512 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.853935 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.854062 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.854164 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.854330 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.854404 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 01:05:11.856184 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:11.849557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.0.97.60:34575)
0619 01:05:11.849739 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:11.849747 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:11.850280 (+ 533us) server_negotiation.cc:407] Connection header received
0619 01:05:11.850470 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:11.850475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:11.850559 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:11.850654 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:11.851365 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.852460 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.854242 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:11.854917 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:11.855041 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:11.855505 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:11.855638 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:11.855739 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:11.855847 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 01:05:11.855916 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:05:11.856797 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22736
I20260619 01:05:11.857846 22798 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:11.969027 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22736
I20260619 01:05:11.989254 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22810
I20260619 01:05:11.990326 22872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:12.094602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22810
I20260619 01:05:12.113968 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 22884
I20260619 01:05:12.115041 22946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:12.225672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 22884
2026-06-19T01:05:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4260 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 01:05:12 dist-test-slave-3rxd krb5kdc[23131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 01:05:12 dist-test-slave-3rxd krb5kdc[23131](info): set up 2 sockets
Jun 19 01:05:12 dist-test-slave-3rxd krb5kdc[23131](info): commencing operation
krb5kdc: starting...
W20260619 01:05:13.303259 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/test-user.keytab.
Jun 19 01:05:13 dist-test-slave-3rxd krb5kdc[23131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T01:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T01:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260619 01:05:13.535215 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37289
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 01:05:13.853590 23144 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:13.853965 23144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:13.854102 23144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:13.854254 23144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:13.861550 23144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:13.861676 23144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:13.861819 23144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:13.861905 23144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:13.862011 23144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:13.871801 23144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:13.873041 23144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:13.874347 23144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 01:05:13.885615 23152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:13.885576 23150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:13.885666 23149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:13.886349 23144 server_base.cc:1034] running on GCE node
I20260619 01:05:13.887120 23144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:13.888365 23144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:13.889564 23144 hybrid_clock.cc:648] HybridClock initialized: now 1781831113889498 us; error 42 us; skew 500 ppm
I20260619 01:05:13.889918 23144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:13 dist-test-slave-3rxd krb5kdc[23131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:13.892849 23144 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260619 01:05:13.894145 23144 webserver.cc:460] Webserver started at http://127.0.97.62:36067/ using document root <none> and password file <none>
I20260619 01:05:13.894779 23144 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:13.894868 23144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:13.895125 23144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:13.897832 23144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data/instance:
uuid: "9a280fce5b3c4c9f98b665824cce2dfa"
format_stamp: "Formatted at 2026-06-19 01:05:13 on dist-test-slave-3rxd"
server_key: "8e373b875466b60be82e5019593c1f96"
server_key_iv: "1fd27aed0a480596109bcf044b117efb"
server_key_version: "encryptionkey@0"
I20260619 01:05:13.898505 23144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a280fce5b3c4c9f98b665824cce2dfa"
format_stamp: "Formatted at 2026-06-19 01:05:13 on dist-test-slave-3rxd"
server_key: "8e373b875466b60be82e5019593c1f96"
server_key_iv: "1fd27aed0a480596109bcf044b117efb"
server_key_version: "encryptionkey@0"
I20260619 01:05:13.903131 23144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 01:05:13.906101 23159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:13.907377 23144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 01:05:13.907496 23144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal
uuid: "9a280fce5b3c4c9f98b665824cce2dfa"
format_stamp: "Formatted at 2026-06-19 01:05:13 on dist-test-slave-3rxd"
server_key: "8e373b875466b60be82e5019593c1f96"
server_key_iv: "1fd27aed0a480596109bcf044b117efb"
server_key_version: "encryptionkey@0"
I20260619 01:05:13.907651 23144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:13.922891 23144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:13.926465 23144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:13.926662 23144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:13.944243 23144 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37289
I20260619 01:05:13.944272 23210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37289 every 8 connection(s)
I20260619 01:05:13.945427 23144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/data/info.pb
I20260619 01:05:13.947569 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 23144
I20260619 01:05:13.947736 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-0/wal/instance
I20260619 01:05:13.948149 388 external_mini_cluster.cc:1428] Setting key a41d11ad7e4c9c21c2047a33731635bc
I20260619 01:05:13.948907 23211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:13.954924 23211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:13 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260619 01:05:13.977089 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:13.957739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37289 (local address 127.0.0.1:45840)
0619 01:05:13.958632 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:13.958695 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:13.960360 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:13.961242 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:13.961246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:13.961762 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:13.962066 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:13.962081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:13.964310 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:13.964313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:13.965305 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:13.965316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:13.966651 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:13.966657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:13.966834 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:13.967454 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:13.967474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:13.970033 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:13.973401 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:13.973409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:13.973417 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:13.973930 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:13.974356 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:13.974359 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:13.974361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:13.974514 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:13.975440 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:13.975446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:13.975639 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 01:05:13.975830 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 01:05:13.976166 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":370,"threads_started":1}
I20260619 01:05:13.977129 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:13.957738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.0.97.62:37289)
0619 01:05:13.959228 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:13.959237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:13.959287 (+ 50us) server_negotiation.cc:407] Connection header received
0619 01:05:13.960715 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:13.960724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:13.961021 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:13.961215 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:13.962272 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:13.964150 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:13.965466 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:13.966412 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:13.966582 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:13.970280 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:13.970289 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:13.970299 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:13.970327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:13.973293 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:13.974038 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:13.974062 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:13.974068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:13.974250 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:13.974643 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:13.974651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:13.974657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:13.975304 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:13.975495 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:13.976010 (+ 515us) server_negotiation.cc:299] Negotiation successful
0619 01:05:13.976342 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":1294,"spinlock_wait_cycles":26880,"thread_start_us":1116,"threads_started":1}
I20260619 01:05:13.980008 23211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:13.980991 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:13.980647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45196)
0619 01:05:13.980807 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:13.980899 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:05:13.981222 23161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
I20260619 01:05:13.982635 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:13.982335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45198)
0619 01:05:13.982460 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:13.982524 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 01:05:13.982820 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260619 01:05:14.002362 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40793
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:05:14.032151 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } attempt: 1
I20260619 01:05:14.033314 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.032803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45200)
0619 01:05:14.033010 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.033125 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 01:05:14.034287 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.033919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45204)
0619 01:05:14.034061 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.034142 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:05:14.034467 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
I20260619 01:05:14.102744 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } attempt: 2
I20260619 01:05:14.104176 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.103511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45212)
0619 01:05:14.103829 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.103964 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 01:05:14.105319 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.104916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45222)
0619 01:05:14.105069 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.105150 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 01:05:14.105558 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
I20260619 01:05:14.196841 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } attempt: 3
I20260619 01:05:14.198261 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.197600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45230)
0619 01:05:14.197908 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.198052 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 01:05:14.199718 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.199118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45236)
0619 01:05:14.199394 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.199500 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260619 01:05:14.200016 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
W20260619 01:05:14.319573 23218 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:14.319923 23218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:14.320052 23218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:14.320192 23218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:14.327349 23218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:14.327435 23218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:14.327491 23218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:14.327607 23218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:14.327700 23218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:14.337347 23218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:14.338665 23218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:14.340041 23218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:05:14.343266 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } attempt: 4
I20260619 01:05:14.344509 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.343924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45242)
0619 01:05:14.344215 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.344345 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 01:05:14.345680 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.345212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45244)
0619 01:05:14.345407 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.345491 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 01:05:14.346014 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40793: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40793: connect: Connection refused (error 111)
W20260619 01:05:14.352092 23223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:14.352138 23226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:14.352180 23224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:14.356051 23218 server_base.cc:1034] running on GCE node
I20260619 01:05:14.356756 23218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:14.358026 23218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:14.359273 23218 hybrid_clock.cc:648] HybridClock initialized: now 1781831114359226 us; error 39 us; skew 500 ppm
I20260619 01:05:14.359580 23218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:14.362500 23218 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260619 01:05:14.363917 23218 webserver.cc:460] Webserver started at http://127.0.97.61:44173/ using document root <none> and password file <none>
I20260619 01:05:14.364590 23218 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:14.364717 23218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:14.365016 23218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:14.367813 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data/instance:
uuid: "fe7b2c6445954d7880bca79a5ed684bc"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "ae20568a8b743e628a1dcb9f41144349"
server_key_iv: "b64938a6455ee9e32eccfe43ebc1d43c"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.368548 23218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal/instance:
uuid: "fe7b2c6445954d7880bca79a5ed684bc"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "ae20568a8b743e628a1dcb9f41144349"
server_key_iv: "b64938a6455ee9e32eccfe43ebc1d43c"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.373420 23218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 01:05:14.376472 23233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:14.377643 23218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 01:05:14.377821 23218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal
uuid: "fe7b2c6445954d7880bca79a5ed684bc"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "ae20568a8b743e628a1dcb9f41144349"
server_key_iv: "b64938a6455ee9e32eccfe43ebc1d43c"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.378018 23218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:14.398617 23218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:14.402079 23218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:14.402328 23218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:14.419898 23218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40793
I20260619 01:05:14.419916 23284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40793 every 8 connection(s)
I20260619 01:05:14.421201 23218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/data/info.pb
I20260619 01:05:14.424089 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 23218
I20260619 01:05:14.424258 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-1/wal/instance
I20260619 01:05:14.424825 388 external_mini_cluster.cc:1428] Setting key 840a7ca0a15e1448a037e1b56b3e6963
I20260619 01:05:14.424925 23285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:14.434353 23285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260619 01:05:14.456099 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.437256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45842 (local address 127.0.97.62:37289)
0619 01:05:14.437483 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.437491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.438608 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 01:05:14.440346 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.440355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.440446 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.440575 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.441937 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.443057 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.444645 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.445411 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.445564 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.449137 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.449144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.449149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.449190 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.452861 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.453518 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.453522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.453524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.453648 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.454138 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.454141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.454143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.454542 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.454705 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.455417 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.455678 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:05:14.456664 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.437654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37289 (local address 127.0.0.1:45842)
0619 01:05:14.438408 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.438458 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.440071 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.440741 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.440748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.441251 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.441756 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.441771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.443218 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.443221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.444492 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.444511 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.445600 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.445604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.445796 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.446454 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.446474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.448921 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.452990 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.452996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.453008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.453415 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.453755 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.453758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.453760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.453991 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.454682 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.454687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.455040 (+ 353us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.455269 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.455617 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":313,"threads_started":1}
I20260619 01:05:14.458400 23285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:14.475525 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.459203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45252)
0619 01:05:14.459494 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.459513 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.459617 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.460678 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.460681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.461062 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.461196 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.461204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.463180 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.463187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.464695 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.464713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.465508 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.465520 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.465747 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.466362 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.466389 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.468790 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.473138 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.473142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.473145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.473506 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.473911 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.473914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.473916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.474036 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.474838 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.474842 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.474960 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.475064 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.475259 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 01:05:14.476044 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.459398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.0.97.61:40793)
0619 01:05:14.460132 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.460141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.460203 (+ 62us) server_negotiation.cc:407] Connection header received
0619 01:05:14.460258 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.460269 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.460499 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.460661 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.461376 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.463020 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.464843 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.465371 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.465460 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.469010 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.469025 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.469045 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.469085 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.472928 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.473640 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.473652 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.473658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.473806 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.474173 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.474180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.474186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.474670 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.474924 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.475246 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.475554 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":392,"threads_started":1}
I20260619 01:05:14.478438 23285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:14.479501 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.479094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46328)
0619 01:05:14.479248 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.479366 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 01:05:14.479733 23235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.481281 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.480951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46344)
0619 01:05:14.481088 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.481188 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:05:14.481491 23285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260619 01:05:14.489249 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-task2ewLDB/build/asan/bin/kudu
/tmp/dist-test-task2ewLDB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39133
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 01:05:14.530834 23285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 1
I20260619 01:05:14.531951 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.531507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46352)
0619 01:05:14.531704 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.531795 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 01:05:14.532912 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.532586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46362)
0619 01:05:14.532743 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.532806 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 01:05:14.533121 23285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.601342 23285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 2
I20260619 01:05:14.602483 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.602028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46364)
0619 01:05:14.602238 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.602334 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 01:05:14.603443 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.603133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46368)
0619 01:05:14.603273 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.603343 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 01:05:14.603639 23285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.645272 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } attempt: 5
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:14.662871 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.646019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45260)
0619 01:05:14.646228 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.646251 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.646381 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.646767 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.646772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.647170 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.647339 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.647348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.648354 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.648359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.649246 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.649257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.650007 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.650012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.650218 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.655078 (+ 4860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.655098 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.657375 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.660412 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.660417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.660421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.660803 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.661173 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.661178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.661182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.661321 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.661865 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.661870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.662033 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.662180 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.662474 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 01:05:14.662871 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.646153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.0.97.61:40793)
0619 01:05:14.646356 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.646362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.646383 (+ 21us) server_negotiation.cc:407] Connection header received
0619 01:05:14.646502 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.646507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.646594 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.646699 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.647495 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.648198 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.649387 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.649849 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.649981 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.657602 (+ 7621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.657607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.657610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.657644 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.660247 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.660928 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.660939 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.660942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.661061 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.661425 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.661432 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.661435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.661742 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.661897 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.662278 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.662511 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 01:05:14.664309 23211 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
I20260619 01:05:14.665272 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.664892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46378)
0619 01:05:14.665067 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.665139 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 01:05:14.666193 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.665880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46382)
0619 01:05:14.666028 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.666094 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 01:05:14.666414 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.694854 23285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 3
I20260619 01:05:14.696043 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.695498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46384)
0619 01:05:14.695737 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.695844 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 01:05:14.697232 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.696706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46396)
0619 01:05:14.696960 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.697068 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260619 01:05:14.697451 23285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.717636 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 1
I20260619 01:05:14.718871 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.718313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46400)
0619 01:05:14.718529 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.718636 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 01:05:14.720067 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.719596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46414)
0619 01:05:14.719778 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.719876 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 01:05:14.720304 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.788508 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 2
I20260619 01:05:14.789718 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.789108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46420)
0619 01:05:14.789366 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.789502 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 01:05:14.790845 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.790417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46430)
0619 01:05:14.790571 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.790658 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 01:05:14.791138 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
W20260619 01:05:14.819744 23292 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 01:05:14.820039 23292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 01:05:14.820108 23292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 01:05:14.820190 23292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 01:05:14.827205 23292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 01:05:14.827289 23292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 01:05:14.827344 23292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 01:05:14.827394 23292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 01:05:14.827451 23292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 01:05:14.837275 23292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37289,127.0.97.61:40793,127.0.97.60:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 18 Jun 2026 23:59:53 UTC on ccycloud.quasar-chlrxl.root.comops.site
build id 4454910
ASAN enabled
I20260619 01:05:14.838466 23292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 01:05:14.839752 23292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 01:05:14.840663 23285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 4
I20260619 01:05:14.841749 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.841314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46436)
0619 01:05:14.841520 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.841611 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 01:05:14.842836 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.842400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46448)
0619 01:05:14.842547 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.842625 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 01:05:14.843061 23285 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
W20260619 01:05:14.852373 23298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:14.852319 23300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 01:05:14.852891 23297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 01:05:14.853266 23292 server_base.cc:1034] running on GCE node
I20260619 01:05:14.854024 23292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 01:05:14.855492 23292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 01:05:14.856730 23292 hybrid_clock.cc:648] HybridClock initialized: now 1781831114856671 us; error 43 us; skew 500 ppm
I20260619 01:05:14.857052 23292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 01:05:14.860105 23292 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260619 01:05:14.861439 23292 webserver.cc:460] Webserver started at http://127.0.97.60:32793/ using document root <none> and password file <none>
I20260619 01:05:14.862155 23292 fs_manager.cc:359] Metadata directory not provided
I20260619 01:05:14.862277 23292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 01:05:14.862591 23292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 01:05:14.865624 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data/instance:
uuid: "fea147a1ab344c0ebe8771a7cbf33c9b"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "93ef0f9ada3435e63a9f5d3d434b5ed3"
server_key_iv: "c0520927633925a40e53d1ecf8956386"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.866446 23292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal/instance:
uuid: "fea147a1ab344c0ebe8771a7cbf33c9b"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "93ef0f9ada3435e63a9f5d3d434b5ed3"
server_key_iv: "c0520927633925a40e53d1ecf8956386"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.871454 23292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 01:05:14.874642 23307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 01:05:14.875990 23292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 01:05:14.876181 23292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data,/tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal
uuid: "fea147a1ab344c0ebe8771a7cbf33c9b"
format_stamp: "Formatted at 2026-06-19 01:05:14 on dist-test-slave-3rxd"
server_key: "93ef0f9ada3435e63a9f5d3d434b5ed3"
server_key_iv: "c0520927633925a40e53d1ecf8956386"
server_key_version: "encryptionkey@0"
I20260619 01:05:14.876384 23292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 01:05:14.894963 23292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 01:05:14.897344 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 3
I20260619 01:05:14.898418 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.897961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46456)
0619 01:05:14.898194 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.898274 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 01:05:14.899013 23292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 01:05:14.899266 23292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 01:05:14.899348 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.899031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46466)
0619 01:05:14.899194 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.899258 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 01:05:14.899616 23211 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39133: connect: Connection refused (error 111)
I20260619 01:05:14.916600 23292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39133
I20260619 01:05:14.916628 23358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39133 every 8 connection(s)
I20260619 01:05:14.917667 23292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/data/info.pb
I20260619 01:05:14.921124 23359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 01:05:14.923362 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-task2ewLDB/build/asan/bin/kudu as pid 23292
I20260619 01:05:14.923550 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task2ewLDB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781830760188020-388-0/minicluster-data/master-2/wal/instance
I20260619 01:05:14.924067 388 external_mini_cluster.cc:1428] Setting key b9c525b0f01e1fcc10b57717696174f9
I20260619 01:05:14.930342 23359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260619 01:05:14.946144 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.925642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46482 (local address 127.0.97.60:39133)
0619 01:05:14.926397 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.926407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.926467 (+ 60us) server_negotiation.cc:407] Connection header received
0619 01:05:14.928022 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.928045 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.928396 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.928684 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.930276 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.931640 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.933234 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.934070 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.934201 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.938611 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.938622 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.938635 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.938677 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.941971 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.942590 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.942598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.942609 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.942821 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.943190 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.943195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.943199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.943735 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.943993 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.944634 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.945052 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":368,"threads_started":1}
I20260619 01:05:14.948474 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.932836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.97.62:37289)
0619 01:05:14.933076 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.933082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.933546 (+ 464us) server_negotiation.cc:407] Connection header received
0619 01:05:14.933745 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.933750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.933816 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.933914 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.935332 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.935958 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.937620 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.938349 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.938489 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.942235 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.942240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.942243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.942273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.945594 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.946439 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.946445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.946448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.946606 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.947063 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.947070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.947074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.947318 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.947480 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.947899 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.948093 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 01:05:14.948511 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.932774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37289 (local address 127.0.0.1:45856)
0619 01:05:14.933421 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.933459 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.933624 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.933987 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.933997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.934834 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.935171 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.935185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.936106 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.936112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.937462 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.937473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.938618 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.938625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.938850 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.939453 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.939477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.942064 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.945738 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.945747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.945753 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.946297 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.946825 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.946829 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.946832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.946968 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.947453 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.947460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.947626 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.947784 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.948096 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":299,"threads_started":1}
I20260619 01:05:14.950838 23359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:14.965215 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.949444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.0.97.62:37289)
0619 01:05:14.949612 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.949618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.949635 (+ 17us) server_negotiation.cc:407] Connection header received
0619 01:05:14.949770 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.949774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.949856 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.949963 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.951406 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.952546 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.954234 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.954837 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.954935 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.959439 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.959444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.959447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.959474 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.962690 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.963398 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.963402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.963404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.963529 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.963903 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.963910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.963912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.964240 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.964512 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.964729 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.964927 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 01:05:14.965658 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.951536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40793 (local address 127.0.0.1:45266)
0619 01:05:14.951824 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.951837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.951959 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.952294 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.952298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.952672 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.952845 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.952857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.954214 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.954221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.955230 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.955244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.956033 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.956041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.956264 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.956730 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.956752 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.959572 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.963142 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.963147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.963152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.963686 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.964083 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.964088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.964091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.964214 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.964799 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.964805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.964960 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.965098 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.965335 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260619 01:05:14.965765 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.951529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.97.61:40793)
0619 01:05:14.951720 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.951726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.951916 (+ 190us) server_negotiation.cc:407] Connection header received
0619 01:05:14.952108 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.952113 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.952182 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.952277 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.953011 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.954006 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.955384 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.955891 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.955982 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.959753 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.959760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.959764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.959803 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.963031 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.963801 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.963804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.963806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.963973 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.964318 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.964326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.964331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.964653 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.964810 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.965205 (+ 395us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.965402 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 01:05:14.966807 23359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Jun 19 01:05:14 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260619 01:05:14.980737 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.967545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.0.97.60:39133)
0619 01:05:14.967688 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.967692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.967706 (+ 14us) server_negotiation.cc:407] Connection header received
0619 01:05:14.967769 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.967771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.967816 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.967881 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.968704 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.969712 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.971369 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.971956 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.972122 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.975267 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.975271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.975274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.975296 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.978134 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.978774 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.978778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.978783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.978907 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.979256 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.979261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.979265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.979581 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.979707 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.980098 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.980437 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":57}
I20260619 01:05:14.980741 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.967398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46488)
0619 01:05:14.967557 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:14.967588 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:14.967678 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:14.967932 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:14.967937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:14.968401 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:14.968581 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.968592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.969833 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.969836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.971241 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:14.971256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.972088 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.972092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:14.972374 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.972859 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:14.972884 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:14.975117 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:14.978330 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.978334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.978336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.978653 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.979000 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:14.979003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:14.979005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:14.979129 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:14.979750 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:14.979759 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:14.979856 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 01:05:14.979994 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 01:05:14.980277 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":59}
I20260619 01:05:14.982209 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:14.966799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.97.61:40793)
0619 01:05:14.966933 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:14.966938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:14.967001 (+ 63us) server_negotiation.cc:407] Connection header received
0619 01:05:14.967169 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:14.967175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:14.967245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:14.967334 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:14.968839 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.969820 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.971478 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:14.972105 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:14.972211 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:14.976595 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:14.976600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:14.976603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:14.976648 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:14.979538 (+ 2890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.980175 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.980180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.980182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.980292 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:14.980732 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:14.980736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:14.980738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:14.981109 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:14.981368 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:14.981638 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 01:05:14.981840 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 01:05:14.986394 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 01:05:14.986498 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: Bootstrap starting.
I20260619 01:05:14.990334 23359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:14.991642 23359 log.cc:824] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:14.989282 23186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9a280fce5b3c4c9f98b665824cce2dfa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45860
I20260619 01:05:14.994428 23359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: No bootstrap required, opened a new log
I20260619 01:05:15.004894 23359 raft_consensus.cc:348] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.005298 23359 raft_consensus.cc:729] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea147a1ab344c0ebe8771a7cbf33c9b, State: Initialized, Role: FOLLOWER
I20260619 01:05:15.005771 23359 consensus_queue.cc:260] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.006603 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.006992 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.007894 23359 sys_catalog.cc:564] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: configured and running, proceeding with master startup.
W20260619 01:05:15.016914 23376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 01:05:15.017033 23376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 01:05:15.014510 23292 master_runner.cc:186] Error getting master registration for 127.0.97.62:37289: OK, instance_id { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" instance_seqno: 1781831113939394 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:05:15.025483 23292 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:05:15.076856 23211 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 4
Jun 19 01:05:15 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:05:15.093029 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.077701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46492)
0619 01:05:15.077918 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:15.077943 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:15.078094 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:15.078456 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:15.078461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:15.078942 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:15.079219 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:15.079236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.080414 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.080422 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:15.081901 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:15.081916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.082658 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.082664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:15.082930 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.083644 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:15.083667 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:15.086514 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:15.090418 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:15.090423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:15.090427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:15.090874 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:15.091254 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:15.091259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:15.091263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:15.091408 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:15.091981 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:15.091986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:15.092150 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 01:05:15.092313 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 01:05:15.092616 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 01:05:15.093053 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.077788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.97.60:39133)
0619 01:05:15.078076 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.078082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.078105 (+ 23us) server_negotiation.cc:407] Connection header received
0619 01:05:15.078218 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.078223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.078304 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.078400 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.079389 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.080269 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.082039 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.082497 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.082642 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.086733 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:15.086744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:15.086751 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:15.086788 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:15.090296 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.090991 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.090996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.091000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.091145 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.091519 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.091524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.091527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.091861 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:15.092030 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.092457 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.092678 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260619 01:05:15.099296 23211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: Bootstrap starting.
I20260619 01:05:15.102360 23211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:15.103430 23211 log.cc:824] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: Log is configured to *not* fsync() on all Append() calls
I20260619 01:05:15.106081 23211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: No bootstrap required, opened a new log
I20260619 01:05:15.111322 23211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.111691 23211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a280fce5b3c4c9f98b665824cce2dfa, State: Initialized, Role: FOLLOWER
I20260619 01:05:15.112191 23211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.113129 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.113507 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.114198 23211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:15.114365 23186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 01:05:15.114603 23186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:15.117921 23186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.120738 23334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b"
I20260619 01:05:15.121204 23334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:15.124615 23334 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a280fce5b3c4c9f98b665824cce2dfa in term 1.
I20260619 01:05:15.125315 23161 leader_election.cc:302] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a280fce5b3c4c9f98b665824cce2dfa, fea147a1ab344c0ebe8771a7cbf33c9b; no voters:
I20260619 01:05:15.125777 23379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 01:05:15.125842 23186 leader_election.cc:288] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [CANDIDATE]: Term 1 election: Requested vote from peers fe7b2c6445954d7880bca79a5ed684bc (127.0.97.61:40793), fea147a1ab344c0ebe8771a7cbf33c9b (127.0.97.60:39133)
I20260619 01:05:15.129555 23379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [term 1 LEADER]: Becoming Leader. State: Replica: 9a280fce5b3c4c9f98b665824cce2dfa, State: Running, Role: LEADER
I20260619 01:05:15.130167 23379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.141544 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a280fce5b3c4c9f98b665824cce2dfa. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.141925 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:15.142258 23285 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } attempt: 5
I20260619 01:05:15.143189 23394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 01:05:15.149176 23395 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260619 01:05:15.151219 23313 tablet.cc:2369] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 01:05:15 dist-test-slave-3rxd krb5kdc[23131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781831114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260619 01:05:15.156013 23394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 01:05:15.160944 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.143545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39133 (local address 127.0.0.1:46500)
0619 01:05:15.143808 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 01:05:15.143831 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 01:05:15.143952 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 01:05:15.144482 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 01:05:15.144487 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 01:05:15.145125 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0619 01:05:15.145322 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:15.145334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.146881 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.146886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:15.148404 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 01:05:15.148419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.150088 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.150094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 01:05:15.150277 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.150987 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 01:05:15.151009 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 01:05:15.153977 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 01:05:15.158188 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:15.158193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:15.158197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:15.158736 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:15.159162 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 01:05:15.159167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 01:05:15.159171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 01:05:15.159338 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 01:05:15.159999 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 01:05:15.160004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 01:05:15.160136 (+ 132us) client_negotiation.cc:769] Sending connection context
0619 01:05:15.160263 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 01:05:15.160547 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
I20260619 01:05:15.161932 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.143115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.97.60:39133)
0619 01:05:15.143329 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.143340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.144093 (+ 753us) server_negotiation.cc:407] Connection header received
0619 01:05:15.144156 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.144161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.144228 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.144310 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.145500 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.146661 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.148593 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.149290 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.149435 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.154184 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:15.154191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:15.154196 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:15.154232 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:15.158035 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.158869 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.158875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.158878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.159034 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.159494 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.159499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.159502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.159820 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:15.159943 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.161357 (+ 1414us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.161600 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 01:05:15.167869 23285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: Bootstrap starting.
I20260619 01:05:15.153612 23144 master_runner.cc:186] Error getting master registration for 127.0.97.61:40793: OK, instance_id { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" instance_seqno: 1781831114414906 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 01:05:15.171657 23144 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 01:05:15.171947 23285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: Neither blocks nor log segments found. Creating new log.
I20260619 01:05:15.173204 23285 log.cc:824] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: Log is configured to *not* fsync() on all Append() calls
W20260619 01:05:15.179281 23165 tablet.cc:2369] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 01:05:15.181470 23285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: No bootstrap required, opened a new log
I20260619 01:05:15.194433 23334 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [term 1 FOLLOWER]: Refusing update from remote peer 9a280fce5b3c4c9f98b665824cce2dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:05:15.195590 23386 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:05:15.214653 23285 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.217823 23285 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe7b2c6445954d7880bca79a5ed684bc, State: Initialized, Role: FOLLOWER
I20260619 01:05:15.218612 23285 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } }
I20260619 01:05:15.225550 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a280fce5b3c4c9f98b665824cce2dfa. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.225849 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.230881 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.231114 23397 mvcc.cc:205] Tried to move back new op lower bound from 7298380247798489088 to 7298380247580000256. Current Snapshot: MvccSnapshot[applied={T|T < 7298380247798489088}]
I20260619 01:05:15.231338 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.233071 23402 mvcc.cc:205] Tried to move back new op lower bound from 7298380247798489088 to 7298380247580000256. Current Snapshot: MvccSnapshot[applied={T|T < 7298380247798489088}]
I20260619 01:05:15.233400 23285 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 01:05:15.233498 23260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fe7b2c6445954d7880bca79a5ed684bc"
I20260619 01:05:15.233762 23259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 01:05:15.238629 23259 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [term 1 FOLLOWER]: Refusing update from remote peer 9a280fce5b3c4c9f98b665824cce2dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 01:05:15.242308 23260 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a280fce5b3c4c9f98b665824cce2dfa in term 1.
I20260619 01:05:15.244518 23379 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 01:05:15.245860 23386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.246238 23386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:15.251931 23365 sys_catalog.cc:455] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.252184 23365 sys_catalog.cc:458] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.254341 23394 catalog_manager.cc:1347] Generated new cluster ID: edb255a538724e8fbe69ad0837082340
I20260619 01:05:15.254437 23394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 01:05:15.271782 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.201644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.0.97.61:40793)
0619 01:05:15.201808 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.201816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.221483 (+ 19667us) server_negotiation.cc:407] Connection header received
0619 01:05:15.222618 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.222626 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.222738 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.222815 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.244888 (+ 22073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.246236 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.248084 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.248794 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.248934 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.252325 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:15.252331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:15.252335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:15.252373 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:15.256411 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.260123 (+ 3712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.260130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.260133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.260283 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.261491 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.261499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.261504 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.261907 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:15.262139 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.270941 (+ 8802us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.271267 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 01:05:15.273305 23215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.196215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.0.97.62:37289)
0619 01:05:15.196497 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.196503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.205769 (+ 9266us) server_negotiation.cc:407] Connection header received
0619 01:05:15.206500 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.206506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.206586 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.206657 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.215480 (+ 8823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.218141 (+ 2661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.230763 (+ 12622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.231551 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.231658 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.251585 (+ 19927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:15.251596 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:15.251608 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:15.251664 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:15.258783 (+ 7119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.260796 (+ 2013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.260801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.260804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.260984 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.262553 (+ 1569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.262558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.262561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.263046 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:15.263350 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.272689 (+ 9339us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.272939 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260619 01:05:15.287045 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.220026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.0.97.60:39133)
0619 01:05:15.220308 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.220314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.230124 (+ 9810us) server_negotiation.cc:407] Connection header received
0619 01:05:15.230250 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.230255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.230326 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.230442 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.248137 (+ 17695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.249146 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.274144 (+ 24998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.274798 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.274902 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.278647 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 01:05:15.278657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 01:05:15.278665 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 01:05:15.278718 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 01:05:15.282828 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.283797 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.283805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.283811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.283969 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 01:05:15.284413 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 01:05:15.284435 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 01:05:15.284438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 01:05:15.284996 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 01:05:15.285294 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.286197 (+ 903us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.286494 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 01:05:15.294356 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a280fce5b3c4c9f98b665824cce2dfa. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.294642 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.295073 23412 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.295451 23412 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc [sys.catalog]: This master's current role is: FOLLOWER
I20260619 01:05:15.296831 23379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a280fce5b3c4c9f98b665824cce2dfa" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37289 } } peers { permanent_uuid: "fe7b2c6445954d7880bca79a5ed684bc" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40793 } } peers { permanent_uuid: "fea147a1ab344c0ebe8771a7cbf33c9b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39133 } } }
I20260619 01:05:15.297111 23379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa [sys.catalog]: This master's current role is: LEADER
I20260619 01:05:15.306838 23394 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 01:05:15.308537 23394 catalog_manager.cc:1504] Loading token signing keys...
I20260619 01:05:15.321463 23394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a280fce5b3c4c9f98b665824cce2dfa: Generated new TSK 0
I20260619 01:05:15.322407 23394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 01:05:15.398404 23425 catalog_manager.cc:1259] Loaded cluster ID: edb255a538724e8fbe69ad0837082340
I20260619 01:05:15.398545 23425 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: loading cluster ID for follower catalog manager: success
I20260619 01:05:15.402449 23425 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: acquiring CA information for follower catalog manager: success
I20260619 01:05:15.404960 23425 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fe7b2c6445954d7880bca79a5ed684bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 01:05:15.489600 23215 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.463143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.97.62:37289)
0619 01:05:15.463349 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.463355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.485348 (+ 21993us) server_negotiation.cc:407] Connection header received
0619 01:05:15.488025 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.488033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.488127 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.488212 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.489358 (+ 1146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45878: BlockingRecv error: recv got EOF from 127.0.0.1:45878 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 01:05:15.494609 23289 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.469722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.0.97.61:40793)
0619 01:05:15.469892 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.469899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.491522 (+ 21623us) server_negotiation.cc:407] Connection header received
0619 01:05:15.492691 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.492697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.492799 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.492880 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.494359 (+ 1479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45302: BlockingRecv error: recv got EOF from 127.0.0.1:45302 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 01:05:15.495973 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.485961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.0.97.60:39133)
0619 01:05:15.486154 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.486161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.492004 (+ 5843us) server_negotiation.cc:407] Connection header received
0619 01:05:15.493124 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.493130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.493211 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.493284 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.495697 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46516: BlockingRecv error: recv got EOF from 127.0.0.1:46516 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 01:05:15.556556 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.540694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.97.61:40793)
0619 01:05:15.540872 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.540878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.548950 (+ 8072us) server_negotiation.cc:407] Connection header received
0619 01:05:15.549295 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.549301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.549496 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.549661 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.550294 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.551538 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.553562 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.554261 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.554355 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.555084 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.555775 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.555885 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.556130 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.556239 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 01:05:15.612752 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.599689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.0.97.60:39133)
0619 01:05:15.599865 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.599872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.610767 (+ 10895us) server_negotiation.cc:407] Connection header received
0619 01:05:15.611131 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.611138 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.611280 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.611377 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.612494 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46532: BlockingRecv error: recv got EOF from 127.0.0.1:46532 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 01:05:15.630594 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.624150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.97.61:40793)
0619 01:05:15.624323 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.624329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.624816 (+ 487us) server_negotiation.cc:407] Connection header received
0619 01:05:15.624969 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.624975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.625074 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.625184 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.625632 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.626739 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.628527 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.629193 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.629319 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.629787 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.629938 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.630029 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.630168 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.630251 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 01:05:15.648757 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.644506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.97.60:39133)
0619 01:05:15.644693 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.644698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.647227 (+ 2529us) server_negotiation.cc:407] Connection header received
0619 01:05:15.647388 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.647394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.647488 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.647614 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.648539 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46546: BlockingRecv error: recv got EOF from 127.0.0.1:46546 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:05:15.660689 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.654354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.0.97.61:40793)
0619 01:05:15.654544 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.654550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.654643 (+ 93us) server_negotiation.cc:407] Connection header received
0619 01:05:15.654922 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.654928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.655053 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.655147 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.655570 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.656791 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.658650 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.659290 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.659405 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.659892 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.660015 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.660108 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.660261 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.660335 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 01:05:15.680900 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.676253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.0.97.60:39133)
0619 01:05:15.676434 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.676440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.679392 (+ 2952us) server_negotiation.cc:407] Connection header received
0619 01:05:15.679604 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.679610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.679702 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.679809 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.680649 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46562: BlockingRecv error: recv got EOF from 127.0.0.1:46562 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:05:15.695026 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.688453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.0.97.61:40793)
0619 01:05:15.688650 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.688657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.689182 (+ 525us) server_negotiation.cc:407] Connection header received
0619 01:05:15.689355 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.689361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.689452 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.689556 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.689968 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.691134 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.692947 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.693664 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.693784 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.694260 (+ 476us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.694402 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.694505 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.694647 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.694758 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 01:05:15.706355 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.704310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.97.60:39133)
0619 01:05:15.704547 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.704553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.705045 (+ 492us) server_negotiation.cc:407] Connection header received
0619 01:05:15.705192 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.705197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.705281 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.705376 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.706113 (+ 737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46564: BlockingRecv error: recv got EOF from 127.0.0.1:46564 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260619 01:05:15.725838 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.719928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.97.61:40793)
0619 01:05:15.720100 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.720106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.720567 (+ 461us) server_negotiation.cc:407] Connection header received
0619 01:05:15.720817 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.720827 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.720920 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.721018 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.721425 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.722295 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.724226 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.724616 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.724697 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.725134 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.725250 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.725314 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.725455 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.725536 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 01:05:15.733525 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.731532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.97.60:39133)
0619 01:05:15.731725 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.731730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.732040 (+ 310us) server_negotiation.cc:407] Connection header received
0619 01:05:15.732232 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.732236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.732305 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.732391 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.733211 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46570: BlockingRecv error: recv got EOF from 127.0.0.1:46570 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 01:05:15.758189 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.752822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.0.97.61:40793)
0619 01:05:15.752991 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.752998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.753321 (+ 323us) server_negotiation.cc:407] Connection header received
0619 01:05:15.753463 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.753469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.753562 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.753663 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.753991 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.755154 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.756356 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.757039 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.757167 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.757531 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.757651 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.757735 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.757852 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.757940 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 01:05:15.765710 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.763939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.97.60:39133)
0619 01:05:15.764154 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.764163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.764370 (+ 207us) server_negotiation.cc:407] Connection header received
0619 01:05:15.764523 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.764527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.764593 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.764672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.765462 (+ 790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46584: BlockingRecv error: recv got EOF from 127.0.0.1:46584 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:05:15.807261 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.801217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.0.97.61:40793)
0619 01:05:15.801482 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.801489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.801828 (+ 339us) server_negotiation.cc:407] Connection header received
0619 01:05:15.801989 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.801995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.802090 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.802222 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.802644 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.803930 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.805183 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.805936 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.806092 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.806396 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.806558 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.806666 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.806820 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.806912 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 01:05:15.816227 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.814090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.97.60:39133)
0619 01:05:15.814302 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.814309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.814682 (+ 373us) server_negotiation.cc:407] Connection header received
0619 01:05:15.814898 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.814907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.815009 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.815126 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.815965 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46594: BlockingRecv error: recv got EOF from 127.0.0.1:46594 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:05:15.889986 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:15.884120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.0.97.61:40793)
0619 01:05:15.884322 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.884327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.884685 (+ 358us) server_negotiation.cc:407] Connection header received
0619 01:05:15.885069 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.885074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.885171 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.885283 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:15.885700 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.886927 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.888159 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:15.888848 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:15.888977 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:15.889289 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:15.889433 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:15.889527 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:15.889619 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 01:05:15.889696 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 01:05:15.897369 23362 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 01:05:15.895600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.0.97.60:39133)
0619 01:05:15.895801 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:15.895806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:15.896081 (+ 275us) server_negotiation.cc:407] Connection header received
0619 01:05:15.896222 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:15.896225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:15.896290 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:15.896416 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 01:05:15.897180 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46610: BlockingRecv error: recv got EOF from 127.0.0.1:46610 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 01:05:16.019157 23376 catalog_manager.cc:1259] Loaded cluster ID: edb255a538724e8fbe69ad0837082340
I20260619 01:05:16.019255 23376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: loading cluster ID for follower catalog manager: success
I20260619 01:05:16.021799 23376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: acquiring CA information for follower catalog manager: success
I20260619 01:05:16.023613 23376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fea147a1ab344c0ebe8771a7cbf33c9b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 01:05:16.033555 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:16.028965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45384 (local address 127.0.97.61:40793)
0619 01:05:16.029129 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:16.029134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:16.029519 (+ 385us) server_negotiation.cc:407] Connection header received
0619 01:05:16.029677 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:16.029681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:16.029745 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:16.029831 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:16.030173 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.030944 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.032207 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.032608 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.032690 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:16.032998 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:16.033099 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:16.033165 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:16.033298 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 01:05:16.033357 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 01:05:16.044318 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:16.038922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.0.97.60:39133)
0619 01:05:16.039128 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:16.039136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:16.039435 (+ 299us) server_negotiation.cc:407] Connection header received
0619 01:05:16.039620 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:16.039630 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:16.039755 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:16.039874 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:16.040260 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.040928 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.042235 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.042608 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.042757 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:16.043274 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:16.043757 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:16.043914 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:16.043979 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 01:05:16.044054 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 01:05:16.056602 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:16.049718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45396 (local address 127.0.97.61:40793)
0619 01:05:16.049943 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:16.049951 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:16.050831 (+ 880us) server_negotiation.cc:407] Connection header received
0619 01:05:16.051096 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:16.051102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:16.051181 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:16.051245 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:16.052038 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.052859 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.054768 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.055214 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.055334 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:16.055843 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:16.055980 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:16.056089 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:16.056246 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 01:05:16.056314 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 01:05:16.057554 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:16.049542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45894 (local address 127.0.97.62:37289)
0619 01:05:16.051067 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:16.051073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:16.051089 (+ 16us) server_negotiation.cc:407] Connection header received
0619 01:05:16.051208 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:16.051216 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:16.051423 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:16.051600 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:16.052391 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.053323 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.055221 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.055692 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.055833 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:16.056556 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:16.056967 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:16.057077 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:16.057203 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 01:05:16.057294 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1395,"thread_start_us":1295,"threads_started":1}
I20260619 01:05:16.058132 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 01:05:16.051035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.0.97.60:39133)
0619 01:05:16.051245 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 01:05:16.051254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 01:05:16.051832 (+ 578us) server_negotiation.cc:407] Connection header received
0619 01:05:16.052130 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 01:05:16.052135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 01:05:16.052232 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 01:05:16.052280 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 01:05:16.052994 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.054035 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.056159 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 01:05:16.056770 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 01:05:16.056885 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 01:05:16.057352 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 01:05:16.057468 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 01:05:16.057522 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 01:05:16.057788 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 01:05:16.057869 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 01:05:16.061651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 23144
I20260619 01:05:16.062583 23206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:16.170076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 23144
I20260619 01:05:16.190740 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 23218
I20260619 01:05:16.191788 23280 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:16.294859 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 23218
I20260619 01:05:16.314008 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task2ewLDB/build/asan/bin/kudu pid 23292
I20260619 01:05:16.315146 23354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 01:05:16.418422 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task2ewLDB/build/asan/bin/kudu with pid 23292
2026-06-19T01:05:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4191 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8452 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356193 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST